Systems and methods for augmented reality based surgical navigation

ABSTRACT

The present disclosure involves object recognition as a method of registration, using a stereoscopic camera on Augmented Reality (AR) glasses or an endoscope as the image capture technology. Exemplary objects include surgical tools, anatomical components or features, such as bone or cartilage, etc. By detecting just a portion of the object in the image data of the surgical scene, the present disclosure may register and track a portion of the patient&#39;s anatomy, such as the pelvis, the knee, etc. The present disclosure also optionally displays information on the AR glasses themselves, such as the entire pelvis, the femur, the tibia, etc. The present disclosure may include combinations of the foregoing features, and may eliminate the need for electromagnetic, inertial, or infrared stereoscopic tracking as the tracking technology.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/853,991 filed May 29, 2019, U.S. ProvisionalPatent Application Ser. No. 62/913,451 filed Oct. 10, 2019, and U.S.Provisional Patent Application Ser. No. 63/000,690 filed Mar. 27, 2020,which applications are hereby incorporated by reference in theirentireties.

BACKGROUND Background Information

Traditional surgical navigation can be broken down into the type oftracking technology used and the type of imaging used, if any.Currently, the most common tracking technologies used for surgicalnavigation are either infrared stereoscopic optical tracking or inertialtracking. Electromagnetic tracking can be used as well but much lessfrequently so now. Infrared stereoscopic optical tracking has thelimitation that the camera needs its own line of site to the surgicalfield and it can only track specific objects that have reflectivespheres that reflect infrared light or have active light emitting diodes(LEDs) that emit infrared light. Such tracking is incapable of seeing,recognizing, and spatially tracking objects.

With respect to imaging, the basic types of navigation are image-basedand image-free. Image-based navigation typically involves using ComputedTomography (CT), Magnetic Resonance (MR) imaging, or 3D Ultrasound andmay include the pre-operative or intra-operative development ofthree-dimensional (3D) models of a patient's anatomy. This computermodel of the patient's anatomy is then matched to the actual patient'sanatomy through a registration process during surgery after a tracker isaffixed to the patient's anatomy. Similarly, navigation analogous toimage-based navigation involves substituting 3D models frompatient-specific imaging with predictive models of the patient, such asstatistical shaped models. For example, a predicted 3D model may begenerated for a patient—as opposed to an actual 3D model for thepatient—based on 2D X-rays of the patient and information from a largedata set of patient statistics and/or statistic shaped models.

For image-free registration, a tracker is similarly affixed to thepatient's anatomy but the anatomy is not registered to a 3D modelderived from imaging. For example, in the case of image-free navigationfor hip arthroplasty, measuring prosthetic acetabular cup orientationand calculating leg length change using image-free navigation techniquesinvolves affixing a tracker to the pelvis. Using one image-free method,the pelvis is then “squared-up”, and that position is set to be thestarting functional coordinate system for the pelvis. Other instrumentsare navigated relative to that.

With a second, more typical image-free prosthetic cup and leg lengthnavigation, a skeletal reference frame (tracker) is affixed to thepelvis and a coordinate system such as the Anterior Pelvic (AP) Planecoordinate system is defined relative to the tracker. The AP Planecoordinate system is defined using a digitizer and entering the twosuperior spine points and the pubic symphysis to instruct the system asto where the tracker is located in space relative to the digitizedcoordinate system.

For image-based registration, after a pelvic tracker is affixed to thepelvis, a digitizer is used to digitize various points on the pelvicbone surface to achieve spatial registration between the computer modelof the patient's pelvis and patient's actual pelvis.

Similarly, the HipXpert® tool from Surgical Planning Associates, Inc. ofMedford, Mass. can be used as a registration and tracking device, aftera pelvic tracker is affixed, by digitizing the three divots on the toolafter the tool is predictably docked to the patient's pelvis. TheHipXpert tool is described in U.S. Pat. No. 8,267,938 for a Method andApparatus for Determining Acetabular Component Positioning, which ishereby incorporated by reference in its entirety.

SUMMARY

Briefly, the present disclosure relates to systems and method forutilizing augmented reality (AR) and/or mixed reality devices to performregistration and/or navigation during surgical procedures. In someembodiments, the AR device may include processors, memory, sensors, andone or more projection systems for displaying virtual images to the userof the AR device, among other elements. Exemplary sensors includephoto/video cameras, depth cameras, light sensors, and microphones,among others. Exemplary images include holograms, e.g., objects madefrom light and sound.

As described, a patient-specific surgical plan may be developed in whichthe locations of surgical tools and/or implants are planned so as toachive one or more goals of the surgery. The planned locations may bedetermined relative to a coordinate system associated with a portion ofthe patient's anatomy, such as the patient's pelvis, femur, tibia,heart, lung, etc. The planned locations also may be translated to berelative to the coordinate system associated with a registration andtracking device that may be affixed to the patient or the plannedlocations may be originally determined relative to the coordinate systemassociated with the registration and tracking device. The systems andmethods may generate virtual images, such as holograms, of theregistration and tracking device, as custom configured for the patient,and of the surgical tools and/or implants at the planned locations.Virtual images of the patient's anatomy or portions thereof may also begenerated. During surgery, with the patient in the operating room,patient registration is performed. In some embodiments, patientregistration is performed using the registration and tracking devicedevice. For example, the hologram of the registration and trackingdevice may be presented and co-located, e.g., aligned, with the physicalregistration and tracking device affixed to the patient in the plannedmanner, for example manually by the surgeon, automatically by thesystems and methods, and/or a combination of manual and automatictechniques. In other embodiments, patient registration may be performedbased on object recognition by the systems and methods of a portion ofthe patient's anatomy, such as recognition of the patient's femoralcondyles, the tibial plateau or the acetabulum as exposed duringsurgery, among other anatomical structures. Holograms of the surgicaltools and/or implants in the planned locations may then be presented,and the physical surgical tools and/or implants may be manipulated,e.g., by the surgeon, to co-locate with the holograms, thereby achievingthe one or more goals of the surgery. In some embodiments, the surgeonmay manually manipulate the hologram of the registration and trackingdevice and/or the physical registration and tracking device or thepatient until the two are co-located. In other embodiments, theregistration tracking device may include a recognizable image, forexample one or more Quick Response (QR) or other codes. The systems andmethods may detect that image, e.g., the one or more QR codes, andautomatically co-locate and anchor the hologram of the registration andtracking device with the physical registration and tracking device. Insome embodiments, the systems and methods may recognize the registrationand tracking device as configured for the patient and docked to thepatient's anatomy, some portion of the patient's anatomy, such as a bonesurface visible through an incision, and/or some combination of QRcodes, registration and tracking device, and patient anatomy. Thesystems and methods may continuously detect the spatial position andorientation of the image, the registration and tracking device, and/orthe patient anatomy during surgery in order to keep the hologramco-located with the physical registration and tracking device.

As noted, in some embodiments, the systems and methods may recognize oneor more objects during surgery. For example, the system and methods mayrecognize some portion of the patient's specific bony anatomy forpatient registration and/or to anchor or co-locate one or more virtualimages, e.g., holograms. In some embodiments, registration of thepatient may be transferred from the registration and tracking device toanother device, e.g., a tracking or anchoring device, allowing removalof the registration and tracking device. As noted, the registration andtracking device may be docked to the patient's anatomy. The tracking oranchoring device may be an implant following implantation, such as aprosthetic cup component implanted in the patient's acetabulum.

Shape data for one or more objects, such of which may bepatient-specific objects may be generated pre-operatively. Exemplaryobjects include anatomical structures, such as the patient's pelvis,acetabulum, femur, tibia, etc., and surgical tools or devices some ofwhich may be customized for the patient, such as tools or devicesadjusted based on the patient's anatomy and templates fabricated tointerfit with the patient's anatomy. The shape data may be in the formof one or more two-dimensional (2D), three-dimensional (3D), or 2D-3Dmodels of the patient-specific object. In some embodiments, the modelsmay be surface models while in other embodiments the models may be solidmodels. One or more coordinate systems may be defined pre-operatively,for example during a planning phase, based on the patient-specificobject. Exemplary coordinate systems include a pelvic coordinate system,a femoral coordinate system, and/or a tibial coordinate system. Thecoordinate systems may be defined automatically, e.g., by a planningtool, manually by a planning surgeon or surgeon's trained associate, orthrough a combination of automated and manual steps. In addition, thelocation of one or more prosthetic components, such as a cup componentand/or a femoral stem component, may be planned relative to the one ormore coordinate systems. The term location may refer to six parametersdetermining the position and orientation of an object in space.

During a planning phase, three-dimensional (3D) models of anatomicalstructures, such as the pelvis, and devices and tools, such as theHipXpert hip registration and tracking device may be generated and usedto plan the surgery for a patient. For example, specific prostheticcomponents may be selected and their locations within the patient's bodydetermined, e.g., to meet one or more goals of the surgery. 3D models ofsurgical tools, such as reamers and cup impactors, may be generated andtheir locations for implanting the selected components at the desiredlocations planned. The desired locations may be final locations, e.g.,of a particular tool, or a sequence of locations, e.g., a tool path,from a starting point of a tool to its final location. At least some ofthe 3D models may be exported into a form that may be used by thehead-mounted AR device to generate respective virtual images. During thesurgical procedure, the surgeon may wear the AR device, which may be anAR head-mounted device (HMD). The AR device may be configured to includeor have access to a navigation system. The navigation system maycooperate with the AR device to generate one or more virtual images,which may be projected onto one or both of the lenses of the AR device,to assist in the surgical procedure. The one or more virtual images maybe in the form of holograms of objects, and the holograms may appearfrom the surgeon's perspective to be in the surgical scene. A hologramis a 3D image formed of light. In some embodiments, the surgeon mayoperate user interface controls to manually resize and move theholograms so that they are co-located with corresponding physicalobjects in the surgical scene. Once co-located by the surgeon, theholograms may be anchored at those locations. The surgeon may thenoperate one or more physical tools until the physical tools areco-located with holograms of the respective tools. With the physicaltools co-located with the holograms of the respective tools, anatomicalstructures may be prepared to receive the prosthetic components asplanned, and the selected components may be implanted at the plannedlocations.

As noted, in some embodiments, a recognizable image, e.g., a QR code,may be affixed to the registration and tracking device in apredetermined location. The systems and methods may detect and recognizethis image, e.g., the QR code. Based on the recognition of the QR code,the systems and methods may co-locate the hologram of the registrationand tracking device to the physical registration and tracking device.Holograms of the surgical tools at the planned locations may then bepresented. In some embodiments, the systems and methods may omitpresenting a hologram of the registration and tracking device andinstead, having recognized the QR code on the physical registration andtracking device, merely present the holograms of the surgical tools atthe planned locations. In some embodiments, multiple QR codes may beused. For example, different QR codes may be placed on the faces of acube mounted to the registration and tracking device. Each QR code mayexpose a spatial coordinate system aligned with the QR code, for exampleat the top left corner of the finder pattern. The AR device may detectthe spatial coordinate system associated with one or more of these QRcodes. The systems and methods may detect the QR code and/or the spatialcoordinate system repeatedly during the surgery, e.g., at some frequencysuch as five times a second, and thus continuously keep the hologramco-located with the physical registration and tracking device. Forexample, the AR device may detect the spatial position and orientationof the image, e.g., QR code(s), the registration and tracking device,and/or the patient anatomy at least periodically over some duration ofthe surgery, such as five times a second or some other frequency,intermittently, continuously, and/or occasionally. The systems andmethods may also use an inertial measurement unit (IMU) to keep thehologram co-located with the physical registration and tracking device,for example if line of sight to the registration and tracking deviceand/or the QR code is lost at any point during the surgery. In someembodiments, the systems or methods may issue one or more alerts and/orwarnings if line of sight to the registration and tracking device and/orQR code has been lost for long enough to risk loss of accurateco-location so that re-anchoring is recommended, which may be apredetermined time. For example, presentation of the hologram of theregistration and tracking device or any other objects or tools may bestopped or suspended until re-anchoring is performed.

At least a portion of the registration and tracking device including theone or more QR codes may be disposed outside of the patient's body. As aresult, the registration and tracking device including the one or moreQR codes may be readily detected by the AR device. Nonetheless, virtualimages, e.g., holograms, anchored based on the detection of theregistration and tracking device may be presented to appear as thoughthey extend into or are entirely disposed inside the patient's body.

In some embodiments, data from the surgical scene as captured by one ormore sensors of the AR device may be processed by the navigation systemthat utilizes the pre-operatively obtained and/or determined shape datafor an object, such as a patient-specific object, to detect the objectin the surgical scene. This may be referred to as an object recognitionmode in which the systems and methods create shape data for an object,such as a patient-specific object, preoperatively and then use objectrecognition techniques to anchor a virtual image to the real object. Itshould be understood that only a portion of the actual object may beobservable in the data captured by the AR device. Nonetheless, thenavigation system may detect the object and determine its location. Thenavigation system may next register the object, e.g., relative to theone or more pre-operatively determined coordinate systems based on thedetection of the object and its determined location. In addition toregistering to a coordinate system, the system, once recognizing andco-locating an object, may display a virtual image of any other objector tool onto the surgical scene in the planned location relative to therecognized object. The navigation system may also track the objectduring the surgical procedure. In some embodiments, registration andtracking of the object may be transferred to a second object, such as atracker placed on the patient.

The navigation system may generate one or more virtual images, e.g.,holograms, which may be projected onto the lenses of the AR device, toassist in the surgical procedure. For example, while only a smallportion of the patient's pelvis or knee may be visible through theincision, a hologram of the entire pelvis may be rendered by the ARdevice and the hologram may be co-located with the patient's physicalpelvis. In other embodiments, holograms of the entire femur and/or tibiamay be rendered and co-located with the patient's femur or tibia, asexamples. Additionally or alternatively, holograms of the one or morecoordinate systems and/or guides for implanting one or more prostheticcomponents at the planned locations may be rendered by the AR device andappear as though they are in the surgical field in order to assist thesurgeon in placing the prosthetic components. In some embodiments, thelocations of the prosthetic components may be changed during thesurgical procedure, and the guides presented to the surgeon by the ARdevice may be updated to conform to these changes. This may be referredto as a live holography mode in which the systems and methodsincrementally or continuously in real time update the holograms toreflect the work performed by the surgical tools, whether directed bythe surgeon or by a robot.

The following outline presents one or more embodiments of the presentdisclosure. It should be understood that different combinations of thesefeatures may be implemented in different embodiments of the presentdisclosure.

-   -   1. Image or object recognition for registration and tracking of        a registration and tracking device, such as the HipXpert tool,        on a patient specific basis. This also registers the pelvis.        Image or object recognition may include at least periodically        detecting and/or recognizing an image or object over some        duration of time during the surgical procedure, such as        intermittently, continuously, and/or occasionally over the        duration of time.        -   1a. augmented reality display of a virtual pelvis            superimposed on the patient's pelvis from the surgeon's            real-time perspective.        -   1b. transferring the pelvic registration to another            recognizable tracking object so that the registration and            tracking tool, e.g., the HipXpert tool, can be removed from            the surgical field.    -   2. Automated registration of the pelvis based on a view of the        acetabulum.    -   3. Combined registration using 1 and 2 to improve the accuracy        of registration. An error in registration can appear visibly as        double vision. Improving the accuracy may reduce or eliminate        such double vision.    -   4. Prepare the acetabulum for total hip arthroplasty (THR), for        example by lining up a physical cup impactor with a hologram of        the cup impactor, perform periacetabular osteotomy, biopsy a        lesion, and/or perform other surgical procedure.        -   a. Track one or more tools used during the procedure and            update the 3D models and/or holograms of the pelvis, femur,            etc. based on what has happened so far in real time.        -   b. Compare three structures during surgery: the original            anatomical structure, the anatomical structure as modified,            and the final goal of how the surgeon wants the anatomical            structure to be modified.    -   5. Automated registration of the femur and tibia for total knee        arthroplasty using object recognition by creating a virtual        patient-specific object, detecting a portion of the real object        within the surgical field, and co-locating and anchoring the        virtual and real objects together both mathematically and        holographically. Registration can be performed using        patient-specific object recognition and either track doing the        same continuously, or switching to another tracking object or        image and tracking of the femur and tibia for total knee        arthroplasty (TKA) or any other femur or tibia intervention that        involves the knee, femur, or tibia. If coordinate systems are        preplanned, then the surgeon may look directly at the ends of        the patient's bones to automatically register the femur and        tibia and start navigating the rest of the surgery right away        without taking the time to perform the traditional registration        steps historically required for surgical navigation. For        example, the systems and methods may determine and present to        the surgeon where the center of the hip is, where the ankle is,        and the coordinate systems of both bones instantly so that he or        she may measure motion, ligament balance, bone resection        details, etc. The surgeon may also navigate all subsequent tools        and show progress of the procedure. The present disclosure may        display augmented reality virtual images projected onto the        patient from the surgeon's exact perspective using a mixed        reality or Augmented Reality (AR) device, such as an AR head        mounted device.    -   6. Embodiments of the present disclosure may transfer        registration from tracking the shape of the end of a bone        (patient-specific object recognition) to another object, such as        a tracker, so that the surgeon can start to modify the bone        surfaces without losing tracking ability.    -   7. Example of endoscopic applications. Using an endoscopic        camera that has stereoscopic vision and/or a depth camera, e.g.,        Time of Flight (ToF) sensors, embodiments of the present        disclosure can register an object using an automated object        recognition as matched to a 3D model of the same object. Then,        if the AR device worn by the surgeon or a stereoscopic tracking        system separate from the AR device located in the operating        room, such as an Infra Red (IR) tracking system, can see a part        of the external portion of the endoscope, the relative location        of the AR device to the endoscope's point of view would allow        the present disclosure to project virtual 3D objects onto the        actual objects from the surgeon's exact point of view. For        example, this may be:        -   a. An endoscopic camera identifies the 3D location of a            human body part using stereoscopy and or a combination of            sensors to achieve automated 3D (object recognition) surface            registration.        -   b. Then, the back end of the endoscopic camera which exits            the person's body can be registered and tracked by the            present disclosure including the AR device and/or the IR            tracking system, among others.        -   c. The AR device may then present virtual images, e.g.,            holograms, of anatomical structures or objects. This allows            the surgeon to “see” through the body and “see” the            structures or objects virtually through the skin or any            other opaque object in between the surgeon and the object.            Optimal locations of ligament placement may be calculated            and presented, e.g., by the AR device, as can optimal tunnel            locations for accessing the calculated ligament placement            locations.

In some embodiments, the present disclosure relates to computer-basedsystems and methods for creating a preoperative plan of a surgicalprocedure and creating one or more holograms that can be presented, forexample during the surgical procedure. The systems and methods includeone or more of a surgical planning system, an Augmented RealityHead-Mounted Display (AR-HMD) configured as a surgical guidance system,and one or more registration and tracking devices. The surgical planningsystem may be utilized to develop a patient-specific surgical plan inwhich the locations of one or more surgical tools, implants, cuttingplanes, drilling axes, etc. may be determined preoperatively so as toachieve one or more goals of the surgical procedure. Additionally oralternatively, the surgical plan may further include plannedmodifications to an anatomical structure, e.g., reshaping a bonesurface. The surgical planning system may generate one or morecomputer-generated models of a portion of a patient's anatomy, such assurface models, based on shape data for the patient from an imagingstudy. The surgical planning system may establish one or more coordinatesystems. The locations of the surgical tools, implants, cutting planesand/or drilling axes and the modifications to the anatomical structuresmay be planned relative to the one or more coordinate systems. In someembodiments, a location of the registration and tracking device(s) mayalso be determined relative to the portion of the patient's anatomy andto the one or more coordinate systems. In some embodiments, thelocations of the surgical tools, implants, cutting planes and/ordrilling axes and the modifications to the anatomical structures may betranslated to a coordinate system for the registration and trackingdevice(s). The planning system may generate images of variouscombinations of one or more of the patient's anatomy, the registrationand tracking device(s), the surgical tools, the implants, the cuttingplanes and/or the drilling axes at the planned locations, and theanatomical structures as modified. The planning system may convert theimages into a format for presentation as holograms by the AR-HMD.

The AR-HMD may utilize image and/or object recognition to recognize theregistration and tracking device(s), an image associated with theregistration and tracking device(s), and/or a portion of the patient'sanatomy to register the patient to the preoperatively generatedholograms. For example, with the patient on an operating table in theoperating room, the registration and tracking device(s) may be docked tothe patient in the planned location (or affixed in a random location).The AR-HMD may detect and track the registration and tracking device(s)during at least a portion of the surgical procedure. The AR-HMD maypresent the holograms and anchor them to the patient based on thecoordinate system for the registration and tracking device(s). Thesurgeon may utilize the holograms as visual guides during the surgicalprocedure. For example, the holograms may be called up and presented ina sequence that follows the steps of the surgical procedure. One or moreholograms may present a surgical tool in a planned location. The surgeonmay manually position the physical surgical tool to be aligned with thesurgical tool of the hologram. One or more holograms may present ananatomical structure modified in a planned manner. The surgeon maymodify the physical anatomical structure to match the holograms. Byusing the holograms as guides for operating surgical tools, modifyinganatomical structures and/or inserting implants, the surgeon may achievethe one or more goals of the surgical procedure.

In some embodiments, the systems and methods do not performintraoperative imaging of the patient and do not track surgical tools orimplants during the surgical procedure. In other embodiments, thesystems and methods may additionally track one or more surgical tools orimplants during the surgical procedure.

BRIEF DESCRIPTION OF THE DRAWINGS

The description below refers to the accompanying drawings, of which:

FIG. 1 is a schematic illustration of an operating room in accordancewith one or more embodiments;

FIG. 2 is a schematic illustration of an Augmented Reality (AR) devicein accordance with one or more embodiments;

FIG. 3 is a pictorial, perspective, exploded view of an AR device inaccordance with one or more embodiments;

FIG. 4 is a pictorial representation of a surgical procedure showing aregistration and tracking device docked on a patient in accordance withone or more embodiments;

FIG. 5 is an illustration of a 3D surface model of a pelvis with a modelof the registration and tracking device docked thereto in accordancewith one or more embodiments;

FIG. 6 is a schematic illustration of an image projected by an AR deviceshowing a virtual image of the patient's pelvis underneath the skin fromthe exact same perspective as the surgeon at that moment in accordancewith one or more embodiments;

FIG. 7 is a pictorial representation of the view into the acetabulum ofa patient through an incision during surgery in accordance with one ormore embodiments;

FIG. 8 is an illustration of a 3D surface model of the patient's pelvisfrom the same perspective as FIG. 7 in accordance with one or moreembodiments;

FIG. 9 is a schematic illustration of an image projected by an AR deviceshowing a virtual image of the patient's pelvis underneath the skin fromthe exact same perspective as the surgeon at that moment in accordancewith one or more embodiments;

FIG. 10 is a pictorial representation of a patient's knee showing a viewof the distal femur during total knee replacement in accordance with oneor more embodiments;

FIG. 11 is an illustration of a 3D surface model of the patient's femurintended to depict the exact same bone in the exact same orientation asthe surgeon's view, for example as determined by automated surfacematching using stereoscopic cameras or any other method of stereoscopicsurface detection in accordance with one or more embodiments;

FIG. 12 is a schematic illustration of an image projected by an ARdevice showing a virtual model of the femur placed in space in the exactsame place as the actual femur as seen from the surgeon's point of viewin accordance with one or more embodiments;

FIG. 13 is a pictorial representation of a patient's knee showing thetibia during total knee replacement in accordance with one or moreembodiments;

FIG. 14 is an illustration of a 3D surface model of the patient's tibiaintended to depict the exact same bone in the exact same orientation asthe surgeon's view in accordance with one or more embodiments;

FIG. 15 is a schematic illustration of an image projected by an ARdevice showing a virtual model of the tibia placed in space in the exactsame place as the actual tibia as seen from the surgeon's point of viewin accordance with one or more embodiments;

FIG. 16 is a schematic, functional illustration of an example navigationsystem in accordance with one or more embodiments;

FIG. 17 is a schematic illustration of an example surgical planningsystem in accordance with one or more embodiments;

FIG. 18 is an illustration of a planning window in accordance with oneor more embodiments;

FIG. 19 is an illustration of a planning window in accordance with oneor more embodiments FIG. 20 is a pictorial representation of a hologramin accordance with one or more embodiments;

FIG. 21 is a pictorial representation of a portion of a registration andtracking tool in accordance with one or more embodiments;

FIG. 22 is a perspective view of a portion of a 3D model of a tool inaccordance with one or more embodiments;

FIG. 23 is an illustration of a planning window in accordance with oneor more embodiments;

FIG. 24 is a pictorial representation of a hologram co-located with aphysical object in accordance with one or more embodiments;

FIG. 25 is a pictorial representation of a hologram in accordance withone or more embodiments;

FIG. 26 is a pictorial representation of a hologram in accordance withone or more embodiments;

FIG. 27 is a pictorial representation of a hologram in accordance withone or more embodiments;

FIG. 28 is a pictorial representation of a hologram in accordance withone or more embodiments;

FIG. 29 is a pictorial representation of a hologram in accordance withone or more embodiments;

FIG. 30 is an illustration of an example planning window for a portionof a surgical plan in accordance with one or more embodiments;

FIG. 31 is a front view of a sizing guide in accordance with one or moreembodiments.

FIG. 32 is a perspective view of a sizing guide in accordance with oneor more embodiments;

FIG. 33 is a front view of a cutting block in accordance with one ormore embodiments;

FIG. 34 is a side view of a cutting block in accordance with one or moreembodiments;

FIG. 35 is a perspective view of a prosthetic knee component inaccordance with one or more embodiments;

FIG. 36 is an illustration of a planning window in accordance with oneor more embodiments;

FIG. 37 is an illustration of a planning window in accordance with oneor more embodiments;

FIG. 38 is an illustration of a planning window in accordance with oneor more embodiments;

FIG. 39 is an illustration of a planning window in accordance with oneor more embodiments;

FIG. 40 is an illustration of a planning window in accordance with oneor more embodiments;

FIG. 41 is a pictorial representation of an example 2D CT image set of apatient's pelvis in accordance with one or more embodiments;

FIG. 42 is a pictorial representation of an example 2D CT image set of apatient's pelvis in accordance with one or more embodiments;

FIG. 43 is a pictorial representation of an example 2D CT image set of apatient's pelvis in accordance with one or more embodiments;

FIG. 44 is a partial side view of a patient's acetabulum with a customfitted template in accordance with one or more embodiments;

FIG. 45 is a perspective view of a portion of a registration andtracking tool in accordance with one or more embodiments;

FIG. 46 is an illustration of a surface model of a pelvis with three cutplanes in accordance with one or more embodiments;

FIG. 47 is an illustration of a surface model of a pelvis with three cutplanes in accordance with one or more embodiments;

FIG. 48 is a pictorial representation of an image generated andprojected by an AR device in accordance with one or more embodiments;

FIG. 49 is an illustration of a surface model of a pelvis illustratingviewpoints of a surgeon in accordance with one or more embodiments;

FIG. 50 is a pictorial representation of an image generated andprojected by an AR device in accordance with one or more embodiments;

FIG. 51 is a pictorial representation of an image generated andprojected by an AR device in accordance with one or more embodiments;

FIG. 52 is a schematic illustration of an operating room in accordancewith one or more embodiments;

FIG. 53 is an illustration of a planning window in accordance with oneor more embodiments;

FIG. 54 is an illustration of cut planes that may be presented by an ARdevice during a surgical procedure in accordance with one or moreembodiments;

FIG. 55 is a pictorial representation of a surgical scene as viewedthrough an AR device in accordance with one or more embodiments;

FIG. 56 is a top view of an example dental model in accordance with oneor more embodiments; and

FIG. 57 is a schematic illustration of a front view of a pelvis inaccordance with one or more embodiments.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

FIG. 1 is a schematic illustration of an operating room 100 inaccordance with one or more embodiments. Disposed in the operating room100 is an operating table 102 on which a patient 104 is positioned for asurgical procedure. Also disposed in the operating room 100 are atracking system 106, a data processing device 110, and a network device,such as a wireless router 112. A surgeon 114 may be in the operatingroom. The surgeon 114 may be wearing an augmented reality (AR) device200, such as a head mounted device (HMD). Optionally, athree-dimensional (3D) detection system 108 may be disposed in theoperating room. Exemplary 3D detection systems include stereoscopiccamera systems, Structured Light imaging systems, and Continuous-Wave(CW) Time of Flight (ToF) imaging systems, such as the Azure KinectDeveloper Kit (DK) from Microsoft Corp. of Redmond, Wash., whichincludes an integrated depth camera, color photo/video camera, inertialmeasurement unit (IMU), and microphone array. The tracking system 106may implement infrared, inertial, or other tracking techniques. The 3Ddetection system 108 may capture images or reflections from object inthe visible or invisible light range. Images generated by the 3Ddetection system 108 may be used in embodiments when the AR device 200includes only a single camera or no cameras. The surgeon 114 maymanipulate one or more surgical tools, such as surgical tool 118. Insome cases, one or more trackers, such as tracker 120, may be attachedto anatomical points of the patient 104. Another tracker 122 may beattached to the surgical tool 118. In some embodiments, the dataprocessing device 110 may host and run some or all of the components ofa navigation system 1600. In some embodiments, some or all of thecomponents of the navigation system 1600 may be run by the AR device200. In some embodiments, other persons in the operating room 100 may bewearing AR devices and holograms presented on the AR device 200 may bepresented on these other AR devices. In some embodiments, one or moredisplay devices may be included in the operating room 100. Imagescaptured by the AR device 200 as well as holograms presented by the ARdevice 200 may be presented on these display devices and watched byothers in the operating room 100 and/or by others observing the surgery.

FIG. 2 is a schematic illustration of an example AR device 200 inaccordance with one or more embodiments. The AR device 200 may includeprojection optics suitable to project a virtual image onto a see-throughor translucent lens, enabling the surgeon 114 to view the surroundingenvironment, such as a surgical field, as well as the displayed virtualimage. The AR device 200 may include a frame 202 having two lenses 204 aand 204 b, two arms 222 a and 222 b, and projectors 208 a and 208 b,which may be disposed on the front of the AR device 200 or in the arms222 a and 222 b, among other places. The projectors 208 a and 208 b mayproject virtual images, e.g., holograms, to the user, for example on thelenses 204 a and 204 b and/or on the user's eyes. The projectors 208 aand 208 b may be nanoprojectors, picoprojectors, microprojectors,femtoprojectors, LASER-based projectors, or holographic projectors,among others. As noted, the two lenses 204 a and 204 b are see-throughor translucent, although in other embodiments only one lens, e.g., lens204 a may be translucent while the other lens 204 b may be opaque ormissing. In some embodiments, the AR device 200 may also include twoarticulating ear buds 220 a and 220 b, a radio transceiver 218, and amicrophone 224. In some embodiments, the AR device 200 may present oneor more sounds associated with holograms and may accept voice commandsfrom the user.

FIG. 3 is a pictorial, perspective, exploded view of the AR device 200in accordance with one or more embodiments. The AR device 200 mayfurther include a plurality of cameras and/or sensors. For example, insome embodiments, the AR device 200 may include a color video camera226, four gray-scale cameras 228 a-d, and one or more depth cameras orsensors, such as a depth camera 230. The AR device 200 also may includeone or more infrared (IR) emitters 232 a-d that work together with thedepth camera 230 as a Continuous-Wave (CW) Time of Flight (ToF)emitter/receiver. The AR device 200 also may include one or moresensors, such as a light sensor 234. It should be understood that the ARdevice 200 may include other sensors, such as accelerometers,gyroscopes, resistive sensors, current sensors, piezoelectric sensors,voltage sensors, capacitive sensors, global positioning satellitereceivers, compasses, altimeters, rangefinders, thermometers, chemicalsensors, eye tracking cameras or sensors, and/or moisture sensors. Insome embodiments, one or more of the sensors may sense movement of thesurgeon 114, such as when and by how much the surgeon 114 moves, tiltsand/or swivels his or her head. For example, a set of sensors may beorganized as an Inertial Measurement Unit (IMU).

In some embodiments, 3D information of the wearer's environment may begenerated from data output by various combinations of the cameras 226,228 a-d, and 230. For example, various combinations of the cameras 226,228 a-d, and 230 may be configured as stereoscopic cameras, a StructuredLight emitter/receiver, or the Continuous-Wave (CW) Time of Flight (ToF)emitter/receiver, among others. Various combinations of the cameras 226,228 a-d, and 230 may be referred to as a spatial detection system.

As described, data output by various combinations of the cameras 226,228 a-d, and 230 included on the AR device 200 may be used to performregistration and/or navigation during one or more surgical procedures.In other embodiments, the AR device 200 may include an infraredstereoscopic tracker. In this case, the AR device 200 may be used toperform infrared stereoscopic tracking of one or more trackers, such asthe tracker 120 and/or tracker 122, among others. Additionally, anaugmented reality viewpoint may be projected onto the AR device 200.

Suitable AR devices include the HoloLens series of mixed reality devicesfrom Microsoft Corp., the Magic Leap One device from Magic Leap, Inc. ofPlantation, Fla., and the Blade smart glasses from Vuzix Corp. of WestHenrietta, N.Y., among others, and are described in U.S. PatentPublication No. 2019/0025587 for AR Glasses with Event and User ActionControl of External Applications to Microsoft Corp. and U.S. PatentPublication No. 2019/0285897 for Display Device to Apple Inc., which arehereby incorporated by reference in their entireties.

FIG. 16 is a schematic, functional illustration of the navigation system1600 in accordance with one or more embodiments. The navigation system1600 may include an object recognizer 1602, an object pose detector1604, an object tracker 1606, a model database 1608, and a virtual imagegenerator 1610. The object recognizer 1602 may include a featuredetector 1612.

It should be understood that the navigation system 1600 is forillustrative purposes only and that the navigation system 1600 may takeother forms including additional and/or other components.

One or more of the components of the navigation system 1600 may beimplemented using computer vision techniques. Alternatively oradditionally, one or more of the components may be implemented usingmachine learning, such as artificial intelligence (AI), techniques.

In other embodiments, some or all of the components of the navigationsystem 1600 may be run on the AR device 200, which as noted may includeone or more processors and memories. In other embodiments, some or allof the components of the navigation system 1600 may be implemented as acloud-based service accessible by a client running on the dataprocessing device 110 and/or on the AR device 200. It should beunderstood that the components of the navigation system 1600 may beimplemented in other ways.

Automated Recognition and Registration of Tools and AnatomicalStructures: Example: The HipXpert Tool

A patient may be diagnosed with a medical condition that requiressurgery. In preparation for the surgical procedure, one or more datagathering procedures may be performed. For example, one or more digitalimages, such as Computed Tomography (CT), Magnetic Resonance Imaging(MRI), conventional radiographs (X-rays), or ultrasonic images, may betaken of the patient. Specifically, images may be taken of that portionof the patient's anatomy on which the surgery is to be performed. Itshould be understood that any diagnostic test or measurement,particularly one that improves dimensional understanding about thespecific portion of the patient's anatomy to be operated upon, may beperformed and used for patient-specific planning.

For example, a patient may be diagnosed with hip joint failure, and mayrequire total hip replacement (THR) surgery either on the left hip, theright hip, or both hips. In this case, one or more CT scans of thepatient's hip may be taken. The one or more digital images (CT,radiographic, ultrasonic, magnetic, etc.) may be taken on the day of thepatient's preoperative visit, at any time prior to surgery, or evenduring surgery. The one or more digital images may providethree-dimensional information regarding the surface and/or structure ofthe patient's hip and associated or adjacent structures.

A surgical planner, such as an experienced surgeon or other person, mayutilize a 3D modeling tool of a planning tool to create one or morecomputer-generated, three-dimensional (3D) models of the patient'sanatomy, such as the patient's hip, based on the one more digital imagestaken of the patient, e.g., CT, MR, or other digital images.Additionally or alternatively to generating a model based on CT, MR, orother digital images, a patient-specific model may be created usingpredictive modeling, e.g., based on patient-specific characteristics.That is, a statistical shaped model or other predictive model may becreated on a patient-specific data input, such as a digital x-ray or acombination of minimum datasets.

The surgical planner may utilize the planning tool to create a surgicalplan for the surgical procedure that is to be performed on the patient.For example, the surgical planner may create a plan for implanting oneor more prosthetic or surgical components, such as an acetabular cupcomponent, into the patient's hip during THR surgery, using one or moresurgical tools. The surgical planner may utilize the planning tool toestablish one or more coordinate systems, such as the anterior pelvic(AP) plane coordinate system, based on the 3D computer-generated modelof the pelvis. Other patient-specific coordinate systems, for example,for use by the one or more surgical tools, may also be established, forexample, by selecting three points on the 3D model of the patient'spelvis, such as an ipsilateral hemipelvic plane coordinate system.Further, “functional” coordinate systems may be established based on theposition of a body part in a functional position. For example, afunctional coordinate system of the pelvis may be established simply byknowing and accepting the position that the patient's pelvis was inwhile the imaging was acquired.

In some embodiments, the surgical planner may utilize the planning toolto calculate one or more inputs and/or adjustments to be made on the oneor more surgical tools, such as the adjustable HipXpert® tool. Theinputs and/or adjustments may be based, at least in part, oninformation, such as spatial information, derived from the 3D model ofthe pelvis that was created, on some or all of the patient-specificinformation, and/or on statistical information known to or accessible bythe surgical planner. For example, the inputs and/or adjustments may beused to customize the HipXpert tool to fit, e.g., dock, to the patient'spelvis, such that the predicted docking location of the HipXpert toolwould be known relative to any other coordinate system of the pelvis,e.g., the AP plane coordinate system. The surgical planner also maychoose particular prosthetic hip components, and may plan their locationwithin the 3D model of the pelvis in order to accomplish a particulargoal for the surgery, such as optimizing the changes in leg length,offset, and/or AP position. In some cases, optimizing the changes maymean minimizing changes to leg length, offset, and/or AP position. Inother cases, it may mean achieving intended changes to leg length,offset, and/or AP position.

The surgical planner may plan the locations of the selected prostheticcomponents achieve the goals. For example, the location of a selectedacetabular cup component within the acetabulum may be determined. Thelocation may include the depth of the cup component in the acetabulumand the planning phase may include determining how the acetabulum shouldbe prepared, e.g., shaped, in order to receive the cup component at theplanned location. For example, the plan may specify the depth and/orshape of the cup bed of the acetabulum. The location may include theorientation of an axis, e.g., a central axis, of the cup componentrelative to the AP plane coordinate system.

A version of the 3D model of the pelvis may be generated with theacetabulum prepared to receive the cup component. For example, a 3Dmodel of the cup bed may be generated. Furthermore, in some embodiments,3D models of the prosthetic components may be included in and/oravailable to the planning tool. The surgical planner may place a 3Dmodel of the cup component at the planned location in the 3D model ofthe pelvis. Similarly, a 3D model of a selected femoral stem may beplaced at the planned location in the 3D model of the hip.

In some embodiments, the HipXpert tool may include a guide, such as arod. The surgical planner may determine one or more adjustments to theHipXpert tool so that, when it is docketed to the patient's pelvis, theguide will point in the direction of acetabular cup orientation, asplanned.

The surgical plan may thus include instructions for setting up and usingone or more surgical tools during the procedure. In other embodiments,the surgical plan may be or may include machine instructions, such asexecutable code, for operating one or more tools or devices, such as asurgical tool or a machine, to assist during the surgical procedure. Insome embodiments, the surgical plan may include machine instructions tobe executed by a robotic surgical tool that will perform all or part ofthe procedure. In addition to controlling a surgical robot, the surgicalplan may provide instructions for controlling a free-hand surgicaldevice, such as a rotating tool, to turn on when it is in a locationwhere cutting is to be performed and either turn off or disable cutting,e.g., through deployment of a protective sheath, when it is in alocation where cutting should not take place.

Exemplary surgical robots include the surgeon-controlled robotic armsfrom Mako Surgical Corp. of Fort Lauderdale, Fla. Exemplary free-handtools include the freehand sculptor from Blue Belt Technologies, Inc. ofPittsburgh, Pa.

Nonetheless, it should also be understood that in some embodiments thesurgical plan may be developed and/or revised during the surgicalprocedure while in other embodiments no explicit surgical plan may becreated. For example, with respect to ACL reconstruction of the knee,one or more statistical shaped models may be used as thepatient-specific shape data and information may be acquiredintraoperatively, such as by landmark digitization and range ofmotion/kinematic assessment, for developing a surgical planintraoperatively.

Manual Registration of Holograms: Example: The HipXpert Tool

As described, during a planning stage, an AP Plane coordinate system maybe defined for a 3D surface model of a patient's pelvis or portionthereof. In some embodiments, a first 3D surface model may include aportion of one or more of the patient's femurs including the femoralheads in the hip joints. A second 3D surface model may omit thepatient's femurs and only include the pelvis or a portion thereof. Insome embodiments, a femoral coordinate system and/or a tibial coordinatesystem may also be defined in addition to the AP Plane coordinatesystem.

FIG. 17 is a schematic illustration of an example surgical planningsystem 1700 in accordance with one or more embodiments. The surgicalplanning system 1700 may include a user interface (UI) engine 1702, amodeling tool 1704, a planning tool 1706, an exporter tool 1708, and adata store 1710. The surgical planning system 1700 may receive patientdata, as indicated at 1712, which may include volume or shape data inthe form of magnetic resonance imaging (MRI) data, computed tomography(CT) data, simultaneous biplanar radiography data, conventional plainradiograph data, ultrasonic data, and/or other data of a patient's hipor other anatomical structure. The surgical planning system 1700 maycreate one or more electronic surgical plans, such as plan 1714, for thehip surgery, and may export one or more files, e.g., for generatingholograms, as indicated at 1716. The surgical planning system 1700 mayinclude or have access to a display 1718.

Suitable tools for generating 2D and/or 3D displays of anatomicalstructures from volume or shape data include the OsiriX image processingsoftware from Pixmeo SARL of Bernex Switzerland, the TraumaCadpre-operative planning system, the MAKOplasty Total Hip Applicationpre-operative and intra-operative planning system, and the HipXpertNavigation System Application 1.4.0. Nonetheless, those skilled in theart will understand that other image processing software may be used.

One or more of the patient data 1712, the surgical plan 1714, and theexported files 1716 may be implemented through one or more datastructures, such as files, objects, etc., stored in the electronicmemory of a data processing device, such as the data store 1710.

As noted, the surgical planner may select one or more prostheticcomponents to be used in a surgical procedure, such as a prosthetic cupcomponent and/or a femoral stem component and plan their placement inthe patient's body. The plan for the prosthetic cup component mayinclude a planned location, including a depth and an orientation withinthe acetabulum. The plan may also include the shape of the cup bed toreceive the cup component. For the femoral stem component, the plan maydefine the location of the femoral stem component within the femur andits orientation relative to the femoral coordinate system and/or tibialcoordinate system.

In some embodiments, the plan may incorporate 3D models of one or moreother tools, such as the HipXpert tool, acetabular reamers and cupimpactors, among others.

FIG. 18 is an illustration of a planning window 1800 generated by thesurgical planning system 1700 and presented on the display 1718 inaccordance with one or more embodiments. The planning window 1800includes a model pane 1802 presenting a 3D model of the patient's pelvis1804. Docked to the model of the pelvis 1804 is a 3D model of theHipXpert tool 1806. As noted, the model of the HipXpert tool 1806 mayinclude a guide, such as a rod 1808. If utilized, the planner maydetermine one or more adjustments to the HipXpert tool so that when itis docked to the patient's pelvis the rod 1808 points in the directionof acetabular cup orientation, as planned.

The surgical planner may plan the position, shape and orientation of thecup bed to receive the prosthetic cup component. FIG. 30 is anillustration of an example planning window 3000 for a portion of asurgical plan in accordance with one or more embodiments. The planningwindow 3000 also includes the model pane 1802 presenting the 3D model ofthe HipXpert tool 1806. A 3D model of a cup bed 3002 as planned may alsobe presented in the model pane 1802. The 3D model of the patient'spelvis appearing in other planning windows may be omitted in theplanning window 3000 for the cup bed 3002. The surgical planner may planthe position, shape and orientation of the cup bed 3002 to achieve thegoals of the surgery. The cup bed refers to the ideal surgically createdbone surface to receive the prosthetic cup component in the plannedlocation.

In some embodiments, the surgical planner may determine the location ofthe acetabular reamer at the 3D model of the pelvis, e.g., relative tothe AP Plane coordinate system, to prepare the cup bed as planned. Forexample, the acetabular reamer may have a handle defining a longitudinalaxis. The surgical planner may position a 3D model of the acetabularreamer so that the cutting basket of the reamer is positioned in theacetabulum to prepare the cup bed as planned in position andorientation.

The surgical planner also may determine the location of the cup impactorat the 3D model of the pelvis, e.g., relative to the AP Plane coordinatesystem, to implant the cup component in the cup bed as planned. Forexample, the cup impactor may have a handle defining a longitudinalaxis. The surgical planner may position a 3D model of the cup impactorso that the longitudinal axis defined by the handle positions the cupcomponent at the end of the cup impactor in the cup bed as planned.

FIG. 19 is an illustration of an example planning window 1900 generatedby the surgical planning system 1700 for a portion of a surgical planand presented on the display 1718 in accordance with one or moreembodiments. The planning window 1900 also includes the model pane 1802presenting the 3D model of the patient's pelvis 1804 and the 3D model ofthe HipXpert tool 1806. A 3D model of a cup impactor 1902 and a 3D modelof a prosthetic cup component 1904 may also be presented in the modelpane 1802. The surgical planner may position the model of the cupcomponent 1904 seated in the cup bed at the planned location andorientation. In addition, the surgical planner may position the model ofthe cup impactor 1902 at the location for implanting the cup component1904 at the planned position and orientation.

FIG. 23 is an illustration of an example planning window 2300 for aportion of a surgical plan generated by the planning system 1700 inaccordance with one or more embodiments. The planning window 2300includes the 3D model of the patient's pelvis 1804 and the 3D model ofthe HipXpert tool 1806. The planning window 2300 further includes a 3Dmodel of a cup component and liner 2302 as implanted in the acetabulumat a desired location, for example relative to the AP Plane coordinatesystem.

In some embodiments, the plan may also include one or more trackingdevices attached to the patient's pelvis whose location is definedrelative to the AP Plane coordinate system or another coordinate system.The one or more tracking devices may include a weathervane type devicethat may be planned to point in the orientation defined for the centralaxis of the prosthetic cup component.

In some embodiments, the plan may include files of 3D models of one ormore of:

the patient's pelvis (or portion thereof);

the patient's femur(s) (both alone and as part of the pelvis);

the HipXpert tool as customized for the patient (both alone and aspositioned on the patient's pelvis);

a reamer tool positioned at the planned depth of the acetabulum and inthe planned orientation for the cup component relative to the AP Planecoordinate system (or a sequence of reamer tools with different size cupreamers leading to a final one);

a hemispherical surface representing the exact position of the ideallyprepared bone surface for receipt of the acetabular component;

a cup impactor tool at the planned position and orientation relative tothe AP Plane coordinate system for the cup component;

the selected prosthetic cup component at the planned orientation anddepth in the acetabulum relative to the AP Plane coordinate system;

the selected prosthetic cup component and liner at the plannedorientation and depth in the acetabulum relative to the AP Planecoordinate system;

the prosthetic stem at the planned orientation and depth relative to thefemoral coordinate system, and/or the tibial coordinate system; and/or

the one or more tracking devices, e.g., weathervane.

It should be understood that various combinations of the above-listed 3Dmodels also may be created.

As described, by anchoring the holograms, the systems and methods do nothave to track any of the surgical tools, e.g., the systems and methodsmay be free of tracking surgical tools. Instead, the surgeon can trackthe instruments using his or her eyes to bring the instruments in linewith the corresponding anchored holograms. Nonetheless, in someembodiments, the systems and methods may track one or more of thesurgical tools.

The planning tool 1706 may export at least some of these 3D model filesinto a format compatible with the AR device 200 so that the AR device200 may project holograms corresponding to the exported 3D model files.For example, one or more of the files representing the 3D objects may beexported and loaded into the memory of the AR device 200. Alternatively,the files representing the 3D objects may be stored at a server and theAR device 200 may be configured as a client capable of accessing thosefiles from the server.

For hip surgery, the following sequence of holograms may be generated:

-   -   1. A hologram of the HipXpert tool and the pelvis;    -   2. A hologram of the HipXpert tool, the pelvis, and the ideal        acetabular cup bed;    -   3. A hologram of the HipXpert tool and the ideal cup bed without        showing the pelvis;    -   4. A hologram of the HipXpert tool, the pelvis, the ideal cup        bed or the cup component, and the acetabular cup component        impaction handle situated in the ideal orientation for        implanting the cup component;    -   5. A hologram of the HipXpert tool, the pelvis, and the metal        acetabular cup component without the bearing insert in which the        native pelvis has all osteophytes still in place, and    -   6. A hologram of the HipXpert tool, the pelvis, the metal        acetabular component, and the bearing insert.

Nonetheless, it should be understood that other and/or additionholograms may be generated and included. Exemplary additional hologramsinclude: holograms of the acetabular reamer handle and each sequentialreamer basket in the ideal location. When the surgeon places the actualreamer handle with the final reamer basket in exact overlap with thehologram of the same, then the cup preparation bed is in the plannedplace. Such additional holograms may have some advantages overabove-described holograms 2 and 3 since the surgeon may be unable to seewhere the reamer is in space when preparing the bony cup bed. Usingthose holograms, the surgeon may have to ream, take the reamer out, andlook into the incision to compare the real prepared bony cup bed surfaceto the hologram. If instead or in addition there is a hologram of theexact reamer handle and basket, the surgeon will be able to tell if thecup bed is correct by looking at overlapping holograms and realitymostly outside of the patient's body. This may be more convenient, amongother advantages. Also, during cup impaction, instead of theabove-described hologram 4 with an idealized straight cup impactor (foralignment only), there may be a hologram of the same exact planned cupimpactor to be used in surgery with the same exact planned cup componentalso to be used in surgery. Then, when impacting the cup, the surgeoncan line up not only the orientation of the cup component to be correct,but can also tell if the cup component is fully seated and if it is inthe correct place.

In some embodiments, computer-generated, three-dimensional (3D) models,such as other Computer Aided Design (CAD) models, of one or moresurgical tools may be stored in the data store 1710. 3D surface modelsof the surgical tools may be generated from these models and also storedin the data store 1710. In some embodiments, only the 3D surface modelsmay be included in the data store 1710. In some embodiments, 3D surfacemodels of one, a handful or some other small number of standard surgicaltools, such as a standard acetabular reamer with a standard cuttingbasket and a standard acetabular cup impactor may be included in thedata store 1710. Holograms that include a reamer or cup impactor may bebased on these surface models of a standard reamer or cup impactor.

However, in other embodiments, 3D surface models for actual reamersand/or cup impactors including entire product families from one or moremanufacturers, e.g., Stryker Corp. of Kalamazoo, Mich., Greatbatch, Inc.(now Integer Holdings Corp.) of Plano, Tex., Ortho Solutions UK Ltd. ofEssex, UK, Zimmer Biomet Holdings, Inc. of Warsaw, Ind., Depuy Synthesof Raynham, Mass., etc., may be included in the data store 1710.Furthermore, 3D surface models for different sizes of cutting basketsand different sizes of acetabular cups may be included in the data store1710. During the surgical planning phase, 3D surface modelscorresponding to the particular reamer and the particular cup impactorthat the surgeon will be using in the surgery may be selected from thedata store 1710 and used in creating the surgical plan. 3D models forcup impactors and cup components may even include spatial assemblyinformation for how each of the planned cup assembles onto the cupimpactor, e.g., due to thread depth and shell thickness). In this way,holograms representing the particular surgical tools that the surgeon isusing may be generated and presented. Furthermore, a sequence ofholograms of a reamer with different basket sizes may be generated toindicate the bone cutting work performed by each reamer basket sizebefore moving to a next reamer basket size. The sequence of hologramsmay illustrate being moved deeper into the acetabulum as further cuttingis performed. That is, each hologram may indicate the exact amount ofcutting to be performed by each reamer basket size. Additionally, ahologram of a cup impactor and cup that corresponds to the physical cupcomponent being implanted may be generated.

Prior to the surgical procedure, the navigation system 1600 or one ormore portions thereof may be loaded into the memory of the AR device 200and/or made accessible to the AR headset 200. For example, the AR device200 may be configured as a client of the navigation system 1600, whichmay be loaded on and run at a server, such as a laptop computer, that isin communicating relationship with the AR device 200. In someembodiments, the planning tool 1706 used to plan the surgery may beloaded and run on the AR device 200.

During the procedure, the surgeon may adjust a physical HipXpert tool asprovided in the plan to customize the tool to fit to the patient'spelvis. The surgeon may then place the physical HipXpert tool on thepatient's pelvis. The patient may be positioned on an operating roomtable. The surgeon may wear the AR device 200. The surgeon may controlthe AR device 200 to render a hologram of the HipXpert tool attached toa hologram of the patient's pelvis as planned. The surgeon may operateuser interface elements provided by the AR device 200 to resize, move,and/or rotate the hologram of the HipXpert tool/pelvis so that thehologram is co-located with the physical HipXpert tool attached to thepatient's pelvis, e.g., aligned together. More specifically, while thepelvis may not be visible to the surgeon because it is below thepatient's skin, the HipXpert tool, which is docked to the patient'spelvis, is visible to the surgeon. Accordingly, the surgeon may resize,move, and/or rotate the hologram of the HipXpert tool/pelvis until it isco-located with the physical HipXpert tool docked to the patient'spelvis. The hologram of the patient's pelvis will also be co-locatedwith patient's pelvis even though the patient's pelvis is not visible tothe surgeon. Once the hologram of the HipXpert tool/pelvis is co-locatedwith the physical HipXpert tool, the surgeon may peg or anchor thehologram of the HipXpert tool/pelvis at that location within theoperating room. For example, the AR device 200 may include an anchoringfeature for holograms rendered by the AR device 200. In addition, asdescribed herein, in some embodiments, the navigation system 1600 mayautomatically co-locate one or more of the holograms with reality, forexample using image recognition of an image, such as a QR code, or usingobject recognition of the HipXpert tool as adjusted specifically for thepatient.

FIG. 24 is a pictorial representation indicated generally at 2400 of ahologram being co-located with a physical object in accordance with oneor more embodiments. The representation 2400 includes a physicalHipXpert tool 2406 docked to a physical hip model 2408 as planned. Therepresentation 2400 further includes a hologram indicated generally at2405 that includes a hologram of a HipXpert tool 2402 and a hologram ofa hip model 2404 in which the HipXpert tool hologram 2402 is docked tothe hologram of the hip model 2404 in the planned manner. The physicalHipXpert tool 2406 includes a QR code 2410. The hologram 2405 may berepositioned in space either manually by the wearer of the AR device 200and/or automatically by the AR device 200 until it is co-located withthe physical HipXpert tool 2406. For purposes of explanation, thepictorial representation 2400 shows the physical hip model 2408.However, a patient's hip will not be visible to the surgeon as it isbeneath the patient's skin. In some embodiments, the surgeon maymanually reposition the hologram 2405 so that the HipXpert tool hologram2402 is co-located with the physical HipXpert tool 2406, which isvisible to the surgeon. While the patient's physical hip is not visibleto the surgeon, the hip hologram (illustrated by the hip model hologram2404) shows the surgeon where the patient's physical hip is. In otherembodiments, the object recognizer 1602 may detect the QR code 2410 onthe physical HipXpert tool 2406 and automatically co-locate the hologram2405 to the physical HipXpert tool 2406. Not only may the objectrecognizer 1602 perform image recognition, such as with a QR code, itmay also perform object recognition of the HipXpert tool 2406 itself orthe HipXpert tool 2406 plus the actual bony acetabulum.

In some embodiments, the physical HipXpert tool may not include a guiderod. Nonetheless, the surgeon may utilize the guide rod of the hologramof the HipXpert tool to implant the prosthetic cup component in thepatient's acetabulum at the planned orientation. That is, the surgeonmay use the guide rod of the hologram of the HipXpert tool as a guidefor implanting the cup at the planned orientation. Nevertheless, inaddition to a hologram of the guide rod (or instead), the AR device 200may present a hologram of the cup impactor tool, and the surgeon mayline up the physical cup impactor tool to this hologram of the cupimpactor tool. The surgeon may then manually line up the physical toolwith the hologram. As described, in some embodiments, it is notnecessary to track the physical tool. Instead, the system may detect oneor more of the QR codes of the HipXpert device and anchor the hologramsbased on the spatial coordinate system exposed by and aligned with theone or more QR codes. The holograms then show the planned locations ofthe surgical tools, and the surgeon may align the physical tool with thehologram, e.g., the planned location for the tool.

In some embodiments, the surgeon may operate the AR device 200 to rendera hologram of the reamer/HipXpert tool/pelvis. The hologram of thereamer may be disposed relative to the hologram of the pelvis such thatthe hologram of the reamer is at the final position and orientation forpreparing the acetabulum to receive the prosthetic cup componentrelative to the AP Plane coordinate system. The surgeon may operate userinterface elements provided by the AR device 200 to resize, move, and/orrotate the hologram of the reamer/HipXpert tool/pelvis so that thehologram is co-located with the physical HipXpert tool attached to thepatient's pelvis, e.g., the hologram and the tool are spatially alignedtogether. The surgeon may operate the AR device 200 to peg or anchor thehologram of the reamer/HipXpert tool/pelvis at that location within theoperating room. The surgeon may then operate a physical reamer tool toprepare the acetabulum until the physical reamer is co-located with thehologram of the reamer. For example, the surgeon may position thephysical reamer to be co-located with the hologram of the reamer. Asnoted, the hologram may represent a standard reamer or, in a preferredembodiment, the hologram may represent the particular reamer being usedby the surgeon in the surgery, which may make it even easier for thesurgeon to line up the physical reamer with the hologram of the reamer.Additionally, a sequence of holograms of reamers, e.g., with differentcutting basket sizes, may be presented, and the surgeon may change thephysical cutting basket to match the cutting basked included in thehologram. The sequence of holograms also illustrates the depth ofcutting to be performed with each cutting basket. When the physicalreamer is lined up with the hologram of the reamer, the cutting by therespective cutting basket is complete. The surgeon may change cuttingbaskets and the next hologram in the sequence may be presented. Thisprocess may be repeated until the cup bed is prepared as planned. Whenthe physical reamer (or the physical reamer with the last cutting basketin the case of a sequence of reamers) is co-located with the hologram ofthe reamer, the cup bed will be prepared for receiving cup component asplanned. Suppose for example, the surgical plan call for a 56 mm cupcomponent. The plan may call for a series of reamers, such as a firstreamer with a 51 mm basket, a second reamer with a 53 mm basket, a thirdreamer with a 55 mm basket, and finally a fourth reamer with a 56 mmbasket to do a final preparation of the cup bed before putting the cupcomponent in.

The surgeon may operate the AR device 200 to render a hologram of thecup bed/HipXpert tool. The surgeon may operate user interface elementsprovided by the AR device 200 to resize, move, and/or rotate thehologram of the cup bed/HipXpert tool so that the hologram is co-locatedwith the physical HipXpert tool attached to the patient's pelvis. Thesurgeon may operate the AR device 200 to peg or anchor the hologram ofthe cup bed/HipXpert tool at that location within the operation room.The surgeon may look through the incision in the patient and compare thephysical acetabulum with the hologram of the cup bed. The surgeon maydetermine whether the appearance of the physical acetabulum followingthe reaming matches the hologram of the cup bed. If not, the surgeon mayoperate the physical reamer to further shape the acetabulum until itmatches the hologram of the cup bed.

FIG. 25 is a pictorial representation of a hologram 2500 in accordancewith one or more embodiments. The hologram 2500 may include the hologram2402 of the HipXpert device, a hologram 2504 of the patient's pelvis,and a hologram 2502 of the cup bed as planned. During the surgicalprocedure, the hologram 2500 may be co-located to the correspondingphysical objects either manually and/or automatically, for example byco-locating the hologram 2402 of the HipXpert device with the physicalHipXpert device. The surgeon may then examine the physical cup bed asprepared, e.g., through the use of the reamer, and see if the shape ofthe physical cup bed, e.g., depth and center or orientation, matches thehologram 2502 of the cup bed as planned. If not, the surgeon maycontinue shaping, e.g., using a reamer, the physical cup bed until itmatches the hologram 2502.

FIG. 26 is a pictorial representation of a hologram 2600 in accordancewith one or more embodiments. The hologram 2600 may include the hologram2402 of the HipXpert device and a hologram 2602 of the prepared cup bedas planned. However, unlike the hologram 2500 (FIG. 25 ), the hologram2600 may not include a virtual representation of the patient's pelvis.During the surgical procedure, the hologram 2600 may be co-located tothe corresponding physical objects either manually and/or automatically,for example by co-locating the hologram 2402 of the HipXpert device withthe physical HipXpert device 2406 (FIG. 24 ). The surgeon may thenexamine the physical cup bed as prepared and see if the shape of thephysical cup bed, e.g., depth and center or orientation, matches thehologram 2602 of the cup bed as planned. It may be easier for thesurgeon to see and compare the physical cup bed with the hologram 2602of the planned cup bed without a virtual representation of the pelvis aswith the hologram 2500, which may interfere with the surgeon's view.Again, if the physical cup bed does not match the shape of the hologram2602 of the planned cup bed, the surgeon may continue shaping thephysical cup bed until it matches the hologram 2602.

Next, the surgeon may operate the AR device 200 to render a hologram ofthe cup impactor/HipXpert tool/pelvis with the cup impactor disposed atthe final location for implanting the prosthetic cup component at theplanned orientation and position, e.g., depth, relative to the AP Planecoordinate system. The surgeon may operate user interface elementsprovided by the AR device 200 to resize, move, and/or rotate thehologram of the cup impactor/HipXpert tool/pelvis so that the hologramis co-located with the physical HipXpert tool attached to the patient'spelvis. The surgeon may operate the AR device 200 to peg or anchor thehologram of the cup impactor/HipXpert tool/pelvis at that locationwithin the operation room.

FIG. 27 is a pictorial representation of a hologram 2700 in accordancewith one or more embodiments. The hologram 2700 may include the hologram2402 of the HipXpert device, the hologram 2504 of the patient's pelvis,the hologram 2602 of the cup bed as planned, and a hologram 2702 of acup impactor disposed at the final location for implanting theprosthetic cup component at the planned orientation and position. Duringthe surgical procedure, the hologram 2700 may be co-located to thecorresponding physical objects either manually and/or automatically, forexample by co-locating the hologram 2402 of the HipXpert device with thephysical HipXpert device.

FIG. 20 is a pictorial representation of a hologram 2000 in accordancewith one or more embodiments. The hologram 2000 may include a hologramof a pelvis 2004, a hologram of the HipXpert tool 2006, and a hologramof a cup impactor 2008. During the surgical procedure, the hologram 2000may be positioned such that the hologram of the HipXpert tool 2006 isco-located, e.g., spatially aligned, with the physical HipXpert tooldocketed to the patient's pelvis. The surgeon may then use a physicalcup impactor 2002 to implant the prosthetic cup component in the cupbed. The surgeon may operate the physical cup impactor 2002 until it isco-located with the hologram 2008 of the cup impactor. When the physicalcup impactor 2002 is co-located with the hologram 2008 of the cupimpactor, the cup component will be positioned in the cup bed asplanned, e.g., at the planned depth and orientation in the acetabulum.

FIG. 55 is a pictorial representation of a surgical scene 5500 as viewedthrough the AR device 200 in accordance with one or more embodiments.Included in the surgical scene 5500 is a patient 5502. Docked to thepatient's pelvis, which is below the skin and not visible, is a threelegged registration and tracking device 5504. The registration andtracking device 5504 includes a cube 5506 with QR codes on its surfaces.Also included in the surgical scene 5500 is a hologram indicatedgenerally at 5508 as presented by the AR device 200. The hologram 5508includes a hologram of the patient's pelvis 5510, a hologram of aregistration and tracking device 5512 and a hologram of a cup impactor5514 at a planned location for implanting a prosthetic cup component. Asillustrated, the hologram of the registration and tracking device 5512is co-located with the physical registration and tracking device 5504,e.g., through image recognition of one or more of the QR codes by the ARdevice 200 or object recognition of at least a portion of theregistration and tracking device 5504. Accordingly, the hologram of thepatient's pelvis 5510 is also co-located with the patient's pelvis. Asurgeon may position a physical cup impactor 5516 in alignment, e.g., beco-located, with the hologram of the cup impactor 5514. While thehologram of the cup impactor 5514 is straight, the physical cup impactor5516, which extends into an incision 5518 and is only partially visible,is C-shaped. With the physical cup impactor 5516 positioned in alignmentwith the hologram of the cup impactor 5514, the surgeon may operate thecup impactor 5516 to implant the cup component disposed at the end ofthe cup impactor 5516 and thus not visible (except through the incision5518) at the planned location.

As described, the systems and methods may register the patient's pelvisduring surgery with the patient in the operating room. Then, a sequenceof holograms may be presented relative to the pelvis as registered. Theholograms may include holograms of surgical tools at planned locationsand the surgeon may line up physical surgical tools with the hologramsto achieve the one or more goals of the surgery. The physical surgicaltools do not themselves have to be tracked in the operating room.Nonetheless, in some embodiments, the surgical tools may be tracked,e.g., by the object tracker 1606.

In some embodiments, in addition to presenting static holograms, the ARdevice 200 may present a sequence of holograms in the form of aholographic movie, which may be paused and resumed by the surgeon asneeded during the surgical procedure. The holographic movie may beupdated, e.g., in real time, for example based on tracking of theoperations of one or more surgical tools.

In some embodiments, the surgeon may operate the AR device 200 to rendera hologram of the prosthetic cup component/HipXpert tool/pelvis with thehologram of the cup component at the planned orientation and locationwithin the acetabulum. The surgeon may operate user interface elementsprovided by the AR device 200 to resize, move, and/or rotate thehologram of the cup component/HipXpert tool/pelvis so that the hologramis co-located with the physical HipXpert tool attached to the patient'spelvis. The surgeon may operate the AR device 200 to peg or anchor thehologram of the cup component/HipXpert tool/pelvis at that locationwithin the operation room. The surgeon may look through the incision inthe patient and compare the location and orientation of the physical cupcomponent with the hologram of the cup component. The surgeon maydetermine whether the appearance of the physical cup component asimplanted matches the hologram of the cup component. If not, the surgeonmay reposition physical cup component until it matches the hologram ofthe cup component.

FIG. 28 is a pictorial representation of a hologram 2800 in accordancewith one or more embodiments. The hologram 2800 may include the hologram2402 of the HipXpert device, the hologram 2504 of the patient's pelvis,and a hologram 2802 of the cup component implanted in the patient'sacetabulum as planned. During the surgical procedure, the hologram 2800may be co-located to the corresponding physical objects either manuallyand/or automatically, for example by co-locating the hologram 2402 ofthe HipXpert device with the physical HipXpert device. The surgeon maythen examine the physical cup component as implanted, e.g., through theuse of the cup impactor, and see if the location and orientation of thephysical cup component matches the hologram 2802 of the cup component asplanned. If not, the surgeon may reposition the physical cup component,e.g., using the cup impactor, until the location of the physical cupcomponent matches the hologram 2802.

In some embodiments, the surgeon may utilize the hologram 2800 todetermine where to insert one or more screws for holding the physicalcup component in place. More specifically, the surgeon may base his orher decision on where to place the one or more screws based on thehologram 2504 of the patient's pelvis. For example, the surgeon mayplace the one or more screws such that they are anchored securely to thepatient's pelvis as indicated by the hologram 2504. For example, the cupmay be planned such that the screw holes in the cup are optimallypositioned to achieve the best fixation with the screws, and the surgeonmay co-locate the physical cup with the hologram during surgery therebyimplementing the planned best fixation.

FIG. 38 is an illustration of an example planning window 3800 generatedby the surgical planning system 1700 and presented on the display 1718in accordance with one or more embodiments. The planning window 3800includes a model pane 1802 presenting a 3D model of the patient's pelvis1804. Docketed to the model of the pelvis 1804 is a 3D model of theHipXpert tool 1806. The pelvis 1804 includes an acetabulum 3802 anddisposed in the acetabulum 3802 is a shell 3804 of an acetabular cupcomponent. The shell 3804 includes a dome hole 3805 for attaching theshell 3804 to a cup impactor and three screw holes 3806 a-c forreceiving bone screws for securing the shell 3804 to the acetabulum3802. The shell 3804 may be rotated within the acetabulum 3802 therebychanging where the screws enter the pelvis. The location of the shell3804 may be planned so that the bone screws will penetrate bone,improving fixation of the screws to the pelvis. The position of thescrew holes 3806 a-c also may be planned so that the bone screws do notextend beyond the bone and injure a blood vessel or other object. Here,the shell 3804 is positioned at minus 20 degrees of rotation. In thislocation, the anterior/inferior screw inserted in the screw hole 3806 cmay have to be short and may even penetrate the anteromedial innercortex, presenting risk to vital structures of the patient.

FIG. 39 is an illustration of an example planning window 3900 generatedby the surgical planning system 1700 and presented on the display 1718in accordance with one or more embodiments. The planning window 3900includes a model pane 1802 presenting a 3D model of the patient's pelvis1804 and the HipXpert device 1806. Here, the shell 3804 is moved to anew location in the acetabulum 3802 relative to the location illustratedin FIG. 38 . Specifically, the shell 3804 is positioned at plus 20degrees of rotation. In this location, the posterior inferior screw hole3806 b is getting closer to where it might need to have a short lengthto avoid extending beyond the posterior wall.

FIG. 40 is an illustration of an example planning window 4000 generatedby the surgical planning system 1700 and presented on the display 1718in accordance with one or more embodiments. The planning window 4000includes a model pane 1802 presenting a 3D model of the patient's pelvis1804 and the HipXpert device 1806. Here, the shell 3804 is moved to anew location in the acetabulum 3802 relative to the location illustratedin FIGS. 38 and 39 . Specifically, the shell 3804 is positioned at zerodegrees of rotation. At this location, all of the screw holes 3806 a-dare in locations that provide excellent screw length supporting strongbone fixation. Accordingly, the planner may choose zero degrees ofrotation for the planned location of the shell during surgery.Furthermore, one or more holograms may be generated based on the modelsof the hip, the HipXpert device, and the shell as illustrated in FIG. 40. The hologram may be presented during surgery and the surgeon may alignthe physical shell with the shell included in the hologram so that thescrew holes are in the planned locations.

In some embodiments, in addition to determining ideal locations for thescrew holes of the shell, the direction and lengths of the bone screwsin the screw holes may also be planned. The direction of the bone screwsmay be planned to maximize screw fixation and/or avoid penetratingbeyond the bone or causing any injury. One or more holograms may begenerated that illustrate the planned directions and lengths of the bonescrews. The representation of the direction of the bone screws may beillustrated in several ways. For example, a line showing the directionsmay be included in the holograms and the surgeon may operate a drill todrill holes for the bone screws along these lines. In other embodiments,holograms of the bone screws at the planned directions with the tips atthe screw holes may be provided. It should be understood that theplanned directions of the bone screws may be illustrated in the hologramin other ways.

In some embodiments, the drilling depth for the bone screws and/or thesize, e.g., length, of each bone screw may be presented in one or moreholograms. For example, a hologram of a drill at the planned depth andwith the drill bit in the planned direction may be presented. Thesurgeon may operate a physical drill so that the physical drill bit isin the planned direction and the surgeon may stop drilling when thephysical drill reaches alignment with the hologram.

This approach for planning bone screws has several advantages. Forexample, it may reduce risk by avoiding dangerous drill trajectories,drilling too far, which might penetrate the far cortex in a dangerouslocation, reduce the risk of placing a screw that is too long in thewrong place, reduce risk by avoiding short screws when longer screws canbe safely placed, and save time since the surgeon need not measure screwdepths during the surgical procedure. It also avoids the risk of usingscrews that are unnecessarily short that would have poor purchase.

With the physical cup component implanted as planned, the surgeon mayinsert a liner into the cup component.

FIG. 29 is a pictorial representation of a hologram 2900 in accordancewith one or more embodiments. The hologram 2900 may include the hologram2402 of the HipXpert device, the hologram 2504 of the patient's pelvis,and a hologram 2902 of the cup component with liner implanted in thepatient's acetabulum as planned. During the surgical procedure, thehologram 2900 may be co-located to the corresponding physical objectseither manually and/or automatically, for example by co-locating thehologram 2402 of the HipXpert device with the physical HipXpert device.The surgeon may then examine the physical cup component with liner asimplanted and see if the location and orientation of the physical cupcomponent with liner matches the hologram 2902. If not, the surgeon mayreposition the physical cup component and/or the liner until itslocation matches the hologram 2902.

Predicted Range of Motion and Impingement.

Preoperatively, the placement of the components and the trimming ofspecific osteophytes can be planned. In addition, range of motion of thehip joint with the planned components and the planned locations may besimulated and the composite range of motion (in all directions) untilsome type of impingement occurs may be calculated. This could be bonefemur-bone pelvis, implant femur-bone pelvis, bone femur-implant pelvis,or implant femur-implant pelvis impingement.

During surgery, once the physical cup is implanted and the physicalosteophytes removed, the AR device 200 may perform object recognition ofthe cup to determine the exact placement of the cup relative to thepelvis. The AR device 200 may determine where the physical cup and/orother implants are, and may further determine the shape of the boneafter osteophyte trimming. The AR device 200 may then update the 3Dsurface model(s) of the pelvis and calculate a range of motion toimpingement based on the location of the cup and/or other implants asimplanted.

During the procedure, the surgeon may check that the physical HipXperttool is still in alignment with the anchored hologram of the HipXperttool. If the surgeon sees that the physical HipXpert tool is no longerco-located with the hologram of the HipXpert tool, the surgeon mayreposition the hologram including the hologram of the HipXpert tool toco-locate the hologram with the physical HipXpert tool and/or mayreposition the patient so that the physical HipXpert tool is co-locatedwith the hologram that includes the hologram of the HipXpert tool. Insome embodiments, the navigation system 1600 may keep the hologramco-located with the physical HipXpert tool automatically, for exampleusing methodologies such as image or object recognition.

In some prior art surgical navigation systems, a surgeon needs to lookaway from the surgical site to a display in order to monitor thetracking of surgical tools. An advantage of the present disclosure isthat the surgeon can keep his eyes trained on the surgical site whiletracking one or more surgical tools.

In some embodiments, the surgeon may attach one or more tracking devicesto the patient. For example, the surgeon may attach a weathervane typedevice or an object with one or more QR codes to the patient's pelvis.The surgeon may operate the AR device 200 to render a hologram of theone or more tracking devices, e.g., the weathervane, the HipXpert tool,and the pelvis. The surgeon may operate user interface elements providedby the AR device 200 to resize, move, and/or rotate the hologram of theweathervane/HipXpert tool/pelvis so that the hologram is co-located withthe physical HipXpert tool attached to the patient's pelvis. The surgeonmay operate the AR device 200 to peg or anchor the hologram of theweathervane/HipXpert tool/pelvis at that location within the operatingroom. The surgeon may adjust the physical weathervane until it isco-located with the hologram of the weathervane. Once the physicalweathervane is co-located with the hologram of the weathervane, thesurgeon may secure or fix the physical weathervane at that location. Thesurgeon may then remove the physical HipXpert device from the patient'spelvis. The surgeon may utilize the physical weathervane and/or thehologram of the weathervane to implant the prosthetic cup component atthe planned orientation and location. For example, the weathervane(physical or hologram) may have an indicator that points along theplanned orientation for the central axis of the prosthetic cupcomponent. The surgeon may use the weathervane (physical or hologram) asa guide to implant the prosthetic cup component at the plannedorientation and/or location.

In some embodiments, the weathervane or a QR cube may be randomlypositioned space in the operating room. The systems and methods couldregenerate new holograms on the fly that show representations of thoseobjects by scanning where they are relative to other objects.

One or more of the holograms described herein may include theweathervane, which may be used as the registration tool in place of orin addition to the HipXpert tool.

With the cup component implanted at the planned location, e.g., depthand orientation, the surgeon may continue with the surgical procedure.For example, the surgeon may reduce the hip joint and close theincision. In other cases, the surgeon may remove the femoral head,implant a prosthetic stem, reduce the hip joint, and close the incision.

In some embodiments, the AR device 200 may utilize object detection todetect the cup component as implanted at the patient's acetabulum. Insome embodiments, the cup component may include a notch or otherphysical feature from which its orientation may be determined by the ARdevice 200. The AR device 200 may register the pelvis based on thelocation of the cup component as detected. The AR device 200 may thenutilize the cup component to anchor one or more holograms as plannedrelative to the pelvis. In some embodiments, once the AR device 200detects the cup component, the HipXpert device may be removed. In otherembodiments, registration of the pelvis may be transferred from the cupcomponent to another object such as a tracker attached to the patient'spelvis. Thus, the AR device 200 may continue to anchor holograms asplanned even if the cup component is no longer in view.

Automated Image Recognition: Example QR Code

In the described embodiments, a surgeon wearing the AR device 200 maymanually register one or more of the holograms to corresponding objectsin the operating room, such as the HipXpert tool.

In some embodiments, the object recognizer 1602 may be configured todetect and track an image, such as a barcode, which may be a twodimensional (2D) Quick Response (QR) code. For example, a QR codetracking tool is available in the Windows Mixed Reality driver forimmersive VR HMDs, such as the HoloLens HMD with the VuForia Engine. Theobject recognizer 1602 may incorporate and/or utilize the Windows MixedReality driver for immersive (VR) HMDs

In some embodiments, one or more QR codes may be added to and/orincorporated into a registration and tracking tool, such as the HipXperttool. The one or more QR codes may be arranged in a predeterminedgeometric relationship relative to the HipXpert tool. For example, athree-dimensional (3D) shape, such as a cube, may be mounted on theHipXpert tool and one or more QR codes may be placed and/or formed onthe respective sides or faces of the cube. The object recognizer 1602may detect at least one of these QR codes, such as the QR code on theside of the cube that faces the AR device 200. Other 3D shapes that maybe used include pyramids, triangular prisms, cuboids, etc.

FIG. 21 is a pictorial representation of a portion of a registration andtracking tool 2100 in accordance with one or more embodiments. The tool2100 may be a HipXpert tool with the compass and guide elements removed.The tool 2100 includes a hub 2102 and two arms 2104 a and 2104 badjustably extending from the hub 2102. The tool 2100 further includesthree (3) legs (not shown) that extend perpendicularly from a nominalplane defined by the hub 2102 and the two arms 2104 a and 2104 b. Afirst leg extends from the hub 2102 and second and third legs extendfrom ends of the two arms 2104 a and 2104 b. Mounted on the hub 2102opposite the legs is a cube 2108. The cube 2108 may include a frontsurface 2110 carrying a QR code 2112. In some embodiments, QR codes maybe placed on more than one side of the cube 2108, such as all but theside used to mount the cube 2108 to the hub 2102, e.g., the bottom side.In addition, the object recognizer 1602 may detect the QR code on theside of the cube 2108 that most closely faces the AR device 200. In someembodiments, the object recognizer 1602 may detect more than one QR codesimultaneously to improve registration and/or tracking accuracy.

As described, the nominal plane of the defined by the hub 2102 and thetwo arms 2104 a and 2104 b may be parallel to the plane defined by thetips of the three legs. When docked to a pelvis, the tips of the threelegs may define a patient-specific ipsilateral hemipelvic plane having aknown geometric relationship to the AP Plane coordinate system for thepelvis. The nominal plane defined by the hub 2102 and the two arms 2104a and 2104 b thus also has a known geometric relationship to the APPlane coordinate system and/or to any other patient-specific coordinatesystems chosen to be defined. Similarly, the cube 2108 is positioned onthe tool 2100 to provide a known geometric relationship between thefront surface 2110 of the cube 2108 which carries the QR code 2112.

A 3D model of the tool 2100 including the cube 2108 and the QR code 2112may be generated.

FIG. 22 is a perspective view of a portion of a 3D model 2200 of aregistration and tracking tool in accordance with one or moreembodiments. The 3D model 2200 corresponds to the physical registrationand tracking tool 2100 including the cube 2108 having the QR code 2112.

In some embodiments, the model of the registration and tracking toolused in the pre-operative planning stage may correspond to the 3D model2200. Similarly, the physical registration and tracking tool used duringthe surgical procedure may correspond to the physical registration andtracking tool 2100. The file(s) of the 3D model 2200 of the tool may beexported to a form from which the AR device 200 may generate one or moreholograms.

During the surgical procedure, the object recognizer 1602 may searchimage or other data captured by the AR device 200 for the QR code(s) onthe physical registration and tracking tool 2100. Upon detecting a QRcode, the object recognizer 1602 may automatically co-locate, e.g.,spatially align, the hologram of the registration and tracking tool withthe physical registration and tracking tool with the QR code. Once thehologram has been co-located with the physical registration and trackingtool 2100, the surgeon may operate the AR device 200 to peg or anchorthe hologram. In this way, the surgeon need not manually co-locate theholograms to the corresponding physical objects/devices. In someembodiments, when the application on the AR device 200 opens, thesurgeon may identify, e.g., point to, a folder created for the patientthat includes all planned holograms in the sequence of the procedure.When the AR device 200 identifies the QR code, a first hologram from thefolder may be displayed in the right scale, position, and orientation.It should be understood that one or more of the holograms do not need toinclude the registration and tracking device itself, e.g., the HipXpertdevice. However, by including the HipXpert tool and the QR cube in theholograms, there is a constant visual confirmation to the surgeon thatthe anchoring is correct, e.g., because the physical HipXpert tool andthe QR code, which sit outside of the patient's body, are co-locatedwith the virtual images of those objects in the hologram.

In some embodiments, one or more applications (apps) may be created andloaded on the AR device 200. The app may include a planning applicationfor running a surgical plan created for a patient and a navigationapplication for detecting a QR code and/or other object and presentingone or more virtual images, e.g., holograms. The app may be controlledthrough user interface elements provided the AR device 200, such as handgestures for opening and interfacing with applications. In otherembodiments, a surgeon may control and/or operate the app using verbalcommands. For example, in response to a first verbal command, e.g.,“load”, the app may automatically open a file explorer window. Thesurgeon can then select a hologram file in a subfolder with a handgesture. The app may automatically pick up a transformation matrix forthe hologram, which may also be located in the same folder, identify thephysical QR code in the surgical scene, and anchor the hologram. Inother embodiments, the surgeon can use other verbal commands to causethe AR device to load and present additional holograms. Exemplary verbalcommands include “hologram2”, “hologram3”, etc. for presenting theholograms in the planned order for the surgical procedure.

One or more components of the navigation system 1600 and/or the surgicalplanning system 1700 may be or may include software modules or librariescontaining program instructions pertaining to the methods describedherein, that may be stored on non-transitory computer readable media,and executed by one or more processors of a data processing device. Insome embodiments, one or more components of the navigation system 1600and/or the surgical planning system 1700 may each comprise registers andcombinational logic configured and arranged to produce sequential logiccircuits. In other embodiments, various combinations of software andhardware, including firmware, may be utilized to implement the presentdisclosure.

In some embodiments, one or more components of the navigation system1600 and/or the surgical planning system 1700 may run on the AR device200. During surgery, the surgeon may open the surgical plan using thesurgical planning system 1700 running on the AR device 200. Asdescribed, the surgical plan may be updated based on the actualalteration of the acetabulum, the femur, or other bone or portion ofanatomy and/or the actual placement of one or more implants.

In some embodiments, the AR device 200 may present one or more of theUser Interfaces of the surgical plan in the operating room for review bythe surgeon. For example, one or more of the User Interfaces may bepresented on a wall or other surface of the operating room.

Transformation Matrices

FIG. 36 is an illustration of an example planning window 3600 generatedby the surgical planning system 1700 and presented on the display 1718in accordance with one or more embodiments. The planning window 3600includes a model pane 1802 presenting a 3D model of the patient's pelvis1804. Docketed to the model of the pelvis 1804 is a 3D model of theHipXpert tool 1806. Mounted on the HipXpert tool 1806 is a cube 3602.The cube 3602 may include a plurality of faces, e.g., surfaces, carryingone or more QR codes, such as a front surface 3604 a, a side surface3604 b, and a top surface 3604 c. One or more coordinate systems may beestablished for the cube 3602. In some embodiments, a coordinate systemmay be established at the center of the cube 3602. For example, anorigin, indicated at 3606 may be located at the center of the cube 3602and x, y and z axes 3608, 3610 and 3612 may be defined relative to theorigin 3606. The x, y and z axes 3608, 3610 and 3612 may be alignedwith, e.g., by parallel to, respective edges of the cube 3602.

In addition, each QR code may expose a spatial coordinate system that isaligned with the QR code, for example at the top left corner of thefinder pattern. As an example, the QR code 3604 b may expose a spatialcoordinate system indicated at 3615. It should be understood that theother QR codes may expose their own spatial coordinate systems. Itshould be understood that the spatial coordinate systems associated withthe QR codes may be aligned at other locations besides the top leftcorner, such as the center of the QR codes, among other locations.

Because the cube 3602 is mounted on the HipXpert device 1806 and theHipXpert device 1806 is docked to the patient's pelvis, the cube 3602 islocated in a fixed location in space relative to the patient's pelvisand thus relative to the AP Plane defined for the patient's pelvis (orany other chosen pelvic coordinate system). In some embodiments, thecube 3602 may always be mounted in the same way to the HipXpert device1806 used for each patient.

In some embodiments, the planning tool 1706 generates onepatient-specific transformation matrix that may be used in determiningwhere to present the virtual images, e.g., holograms, created for asurgical procedure. For example, the planning tool 1706 may generate apatient-specific transformation matrix that determines the orientationand position of the virtual image, e.g., hologram, relative to thecoordinate system established at the center of the cube 3602. Inparticular, the transformation matrix may specify the orientation andposition of the hologram relative to the coordinate system that includesthe origin 3606 and the x, y and z axes 3608, 3610 and 3612 defined forthe front face 3604 a of the cube 3602. This patient-specifictransformation matrix may relate the coordinate system at the center ofthe cube to the random position of the patient in the CT scanner (orother image modality) from which the surface models of the patient'sanatomy are generated.

In addition, a transformation matrix may be defined that relates thespatial coordinate system associated with each QR code to the coordinatesystem established at the center of the cube 3602. Because it is a cube,these transformation matrices may all be the same.

As described herein, during the surgical procedure, the AR device 200may detect the QR code applied to one of the faces or surfaces of thephysical cube mounted on the physical HipXpert device that is docked tothe patient's pelvis. The AR device 200 may utilize the transformationmatrix associated with the detected QR code and the patient-specifictransformation matrix to orient and position the virtual image, e.g.,the hologram. The AR device 200 may anchor the hologram relative to thecoordinate system at the center of the cube. In some embodiments, thepatient-specific transformation matrix may be stored in the folder withthe holograms. The transformation matrix or matrices associated with theQR codes may be hard coded in the application or in other embodimentsmay also be stored in the folder. When the AR device 200 accesses ahologram from the folder for presentation, the AR device 200 may alsoretrieve the patient-specific transformation matrix.

As noted, a patient-specific transformation matrix may be defined forthe holograms that will be presented during a surgical procedure. Thispatient-specific transformation matrix may be defined relative to aselected point of the cube 3602. The selected point may be the center ofthe cube 3602. As noted, the cube 3602 may be mounted to the HipXpertdevice, which in turn is docked to the patient's pelvis in apredetermined and known location. Accordingly, the center of the cube3602 is in a fixed and known location relative to the patient's pelvis,e.g., relative to the AP Plane (or any other pelvic coordinate system).Locations and orientations of implants, e.g., the cup component, andtools, e.g., reamers and cup impactors, may be planned for a patient,e.g., relative to the AP Plane. Geometric relationships between theseplanned locations and orientations and the center of the cube 3602 maybe determined. During surgery, the AR device 200 may recognize one ormore of the QR codes on the physical cube of the HipXpert as docked tothe patient. With the location of the physical cube in space determined,the AR device 200 can then use the patient-specific transformationmatrix to determine where to locate the holograms such that theholograms appear in the planned locations and orientations.

In some embodiments, one or more secondary transformation matrices mayalso be defined. For example, secondary transformation matrices may bedefined for each of the five QR codes applied to the faces of the cube3602, e.g., front face, left face, right face, rear face, and top face.These secondary transformation matrices may provide geometric transformsfrom the respective QR code to the patient-specific primary matrixdefined for the center of the cube 3602. When the AR device 200 detectsa QR code (the particular QR code depending on the way the surgeonhappens to be viewing the HipXpert device), the AR device 200 mayretrieve the secondary transformation matrix associated with thedetected QR code. The AR device 200 may then utilize this secondarytransformation matrix together with the patient-specific transformationmatrix to orient and position the respective hologram. While thetransformation matrix generated for the center of the cube 3602 may bepatient-specific, the secondary transformation matrices are notpatient-specific. Instead, the secondary transformation matrices are thesame for each cube geometry, e.g., dimensions. Thus, assuming the samecube 3602 is being reused or a cube 3602 with the same dimensions isbeing used with another patient, the same secondary transformationmatrices may be re-used.

In sum, just a single patient-specific transformation matrix between theorientation and position of the QR code and the orientation and positionof the rest of the hologram for every hologram that is to be presentedmay be generated. With the present disclosure, by detecting in space aQR code (that is on a cube mounted on a HipXpert device docked to apatient's pelvis), the AR device 200 can automatically register andtrack the patient's pelvis and allows for the presentation of one ormore co-located holograms. In particular, the tips of the legs of theHipXpert device when docked to a patient's pelvis may define ahemi-pelvic ipsilateral reference plane having a known geometricrelationship to the AP Plane. Furthermore, the frame of the HipXpertdevice from which the legs extend may be parallel to this hemi-pelvicipsilateral reference plane (and thus have a known geometricrelationship to the AP Plane). The cube which carries the one or more QRcodes may be mounted on this frame. Accordingly, by detecting a QR code,the pelvis may be registered and tracked.

FIG. 37 is an illustration of an example planning window 3700 generatedby the surgical planning system 1700 and presented on the display 1718in accordance with one or more embodiments. The planning window 3600includes a model pane 1802 presenting a 3D model of the patient's pelvis1804. Docketed to the model of the pelvis 1804 is a 3D model of theHipXpert tool 1806. Mounted on the HipXpert tool 1806 is the cube 3602.An AP Plane 3702 is defined for the pelvis 1804.

As described, the QR cube may be mounted on a central portion of theframe of the HipXpert device. Because the legs of the HipXpert devicemay be of fixed lengths, the location of the QR cube and thus QR code(s)is constant from one patient to another. A patient-specifictransformation matrix instructs the system as to where the QR cube andQR code(s) are located in space relative to random image-spacecoordinate system and also the anterior pelvis plane coordinate system.This transformation matrix is then a predetermined “patient-specificpass-code”. When the holograms are exported, the “key” or patientspecific transformation matrix is also exported, which is used todetermine where to present the holograms in space for that patient'sspecific surgical plan.

Cross-Section Display of Image Data Such as CT or MR Data.

As described, images of a patient such as a CT or MR study may be takenof a patient during a preoperative phase. For example, for hip surgery,a CT scan may be taken of the patient's pelvis and hips (with someimages of the distal femur for coordinate system development). For kneesurgery, a CT scan or MR study may be taken of the patient's knee(again, potentially with images of the hip and ankle for coordinatesystem development). Such image modalities create an image volume thatcan be displayed as sequential sliced in the original image acquisitionplane, or can be displayed in any cut plane through the image volume. Infact, the display need not be a perfect plane, the image sampling couldbe made in any desired shape. For the purposes of this discussion theimages could be generated as planar images. In addition, the imagevolume may be used to construct a 3D surface model, e.g., of thepatient's pelvis or knee. The 3D surface model may be opened andmanipulated using a CAD software environment. Pre-operative planning maybe performed using the 3D surface model. For example, the 3D surfacemodel may be used to plan the preparation of bone surfaces and theselection, location and orientation of one or more prosthetic implants.

In some embodiments, the entire image data volume such as a CT imagevolume for a patient or a portion thereof may be loaded onto orotherwise made accessible to the AR device 200. During surgery, the ARdevice 200 may display desired sub-sections of the image volume to thesurgeon. For example, the AR device 200 may register the portion of thepatient's anatomy being operated on using one of the registrationmethods described, such as the patient's pelvis or knee, and thentracked using a registration and tracking device such as a QR cube asdescribed. The AR device 200 may then co-locate and anchor the entireimage volume, such as a CT data volume, in space relative to theregistration and tracking device. The system then may give the surgeonthe option of seeing a portion of the image volume in space co-locatedwith the actual location that the image data was acquired from on thepatient. For example, the image volume could be cut in a planarcross-section that is perpendicular to the view of the surgeon wearingthe AR device 200. That planar cross section could be determined as afixed distance from the viewer or for example a fixed origin within thevolume. For example, the surgeon, when preparing the acetabulum, maywant to know the thickness of the remaining bone deep to the proposedcup placement. The origin of the cross section could be fixed at thecenter of the proposed placement of the acetabular component, and thedisplayed planar section through the volume would vary as the surgeonmoves to stay perpendicular to the viewpoint of the surgeon's eyes.

For example, the CT data volume for the patient's pelvis may beco-located with the patient's pelvis in the operating room. The ARdevice 200 may generate one or more planar cuts through the CT datavolume to produce a two dimensional (2D) CT image from the CT data. TheAR device 200 may present this 2D CT image to the surgeon. The 2D CTimage may be generated from a planar cut, also referred to as a cutplane, through a plurality of the slices included in the CT data volume.The planar cut through the CT data volume may be perpendicular to thesurgeon's line of sight relative to the CT data volume as co-locatedwith the patient's anatomy, e.g., the pelvis or knee. By co-locating theCT data volume with the patient, the 2D CT image, as displayed by the ARdevice 200, may appear to the surgeon as overlaid on and co-located withthe patient's anatomy. The cut plane may be set at a predetermineddistance from the AR device 200. For example, if the surgeon moves hisor her head and consequently the AR device 200 closer to the patient(e.g., lying supine on the operating table), the cut plane is movedbackward (posterior) through the CT data volume. Similarly, as thesurgeon moves his or her head away from the patient, the cut plane movesforward (anterior) through the CT data volume. Thus, by simply movinghis or her head, the surgeon can control where the cut plane is formedin the CT data volume, and thus the resulting 2D CT image generated andpresented by the AR device 200.

FIG. 41 is a pictorial representation of an example 2D CT image set 4100of a patient's pelvis in accordance with one or more embodiments. The 2DCT image set 4100 may include an image 4102 through an axial plane, animage 4104 through a coronal plane, and an image 4106 through a sagittalplane. The coronal image 4104 shows the patient's left and right hipjoints and a portion of the patient's spine. Suppose the patient islying supine on an operating table, and the surgeon is looking down atthe patient. The AR device 200 may generate and present a 2D CT imagesimilar to the image 4104 through the coronal plane. The 2D CT image maybe formed based on a cut plane indicated at 4108 on the sagittal image4106 that is a predetermined distance from the AR device 200.

Now, suppose the surgeon moves his or her head away from the patient.

FIG. 42 is a pictorial representation of an example 2D CT image set 4200of a patient's pelvis based on the new position of the surgeon's head inaccordance with one or more embodiments. The 2D CT image set 4200 mayinclude an axial image 4202, a coronal image 4204, and a sagittal image4206. As illustrated, because the surgeon moved his or her head awayfrom the patient, the cut plane 4208, which remains a fixed distancefrom the AR device 200, is moved anterior through the CT data. Thecoronal image 4204 is thus different than the coronal image 4104 (FIG.41 ).

Now, suppose the surgeon moves his or her head closer to the patientrelative to the distance producing the 2D CT image set 4100 (FIG. 41 ).

FIG. 43 is a pictorial representation of an example 2D CT image set 4300of a patient's pelvis based on the new position of the surgeon's head inaccordance with one or more embodiments. The 2D CT image set 4300 mayinclude an axial image 4302, a coronal image 4304, and a sagittal image4306. As illustrated, because the surgeon moved his or her head closerto the patient, the cut plane 4108, which remains a fixed distance fromthe AR device 200, is moved posterior through the CT data volume. Thecoronal image 4304 is thus different than the coronal images 4104 (FIG.41 ) and 4204 (FIG. 42 ).

As noted, the 2D CT image generated and presented by the AR device 200may be based on a cut plane that is a fixed distance from the AR deviceand perpendicular to the surgeon's line of sight. A suitable fixeddistance is 50 cm for example. The 2D CT image is thus a cross-sectionof the CT data volume. In other embodiments, the 2D CT image data maycorrespond to one of the slices of the CT data volume.

In some embodiments, the AR device 200 may present one or more hologramsin addition to the 2D CT image. For example, in addition to the 2D CTimage, the AR device 200 may present one of the holograms including thereamer tool, the cup impactor tool, a cup component, a knee cutting jig,a knee component, etc. The presentation of one or more 2D CT imagestogether with a hologram of a reamer may provide the surgeon withadditional information, such as whether the reamer is getting close toreaming all the way through the inner wall of the patient's acetabulum.For example, while the AR device 200 presents a 2D CT image, the surgeoncould intuitively determine how far the reamer has cut into thepatient's acetabulum, e.g., by placing his or her finger in the woundwhile viewing the 2D CT image.

As the surgeon is reaming the acetabulum to prepare the cup bed forreceiving the cup component, he or she may want to know how much bone isleft behind the reamer medially, for example to avoid going through thebone. A cut plane that is along the surgeon's line of sight whilereaming would provide this information. In some embodiments, the ARdevice 200 may present such a cut plane through the CT volume data. Thecut plane display may be locked in position so that the surgeon may thenmove his or her head to observe the cut plane and thus see how much boneis left behind the reamer. In other embodiments, another medicalprofessional in the operating room wearing an AR device 200 may observethis cut plane and inform the surgeon of how much bone is remaining.

FIG. 46 shows a surface model of the pelvis 4602 with 3 cut planes. Thegreen box 4604 signifies one image-generation plane, the red box 4606signifies a second image-generation plane, and the yellow box 4608signifies a third image-generation plane.

FIG. 47 shows a purple arrow 4702 pointing to a particular red arrow4704 from the same image as illustrated in FIG. 46 . A surgeon mightoften view the hip from the perspective of the designated red arrow4704.

FIG. 48 is a pictorial representation of an image 4800 projected by theAR device 200 in the exact location within the patient's body that thedata were acquired from. This image 4800 represents an image generatedin the yellow box plane 4608 of FIG. 46 in that it is both perpendicularto the surgeon's viewpoint and in a plane that includes the center ofthe planned acetabular component. FIG. 48 also shows a cross section ofthe planned acetabular component indicated at 4802 that could be turnedon or off depending upon the surgeon's preference.

FIG. 49 shows the original surgeon's viewpoint (the red arrow 4704designated by the purple arrow 4702) and a potential second viewpointthat is the red arrow 4902 designated by the light blue arrow 4904.

FIG. 50 is a pictorial representation of an image 5000 generated by theAR device 200 of a cut plane in the plane of the green box 4604, beingperpendicular to the surgeon's line of sight when viewing from the pointof view of the red arrow 4902 that is designated by the light blue arrow4904. The AR device 200 may display the image 5000 in the exact locationfrom which the image pixels were acquired from inside the patient's bodyat the time that the CT study (or any other image study with such adataset) was acquired.

As described, the AR device 200 may automatically display images thatare perpendicular to the surgeon's viewpoint in real time as the surgeonmoves his or her head around. The AR device 200 also may “hold” thedisplay of an image in the green box 4604, e.g., in response to userinput, and the surgeon wearing the AR headset 200 may be able to movethe AR headset 200 around without causing a new image to berecalculated.

The AR device 200 may thus create and present images that are co-locatedwith the actual patient, from any desired angle, depth, and shape. Inaddition, the image need not even be a planar image.

FIG. 51 is a pictorial representation of an image 5100 generated by theAR device 200 of a cut plane in the plane of the red box 4606.

In some embodiments, multiple planar cuts may be made through the CTvolume data and presented by the AR device 200. For example, threeorthogonal, planar cuts can be made in the CT volume data and presentedby the AR device 200.

It also should be understood that the cuts made through the CT volumedata need not be planar. For example, a curved cut or other shaped cutmay be made through the CT volume data and presented by the AR device.

In addition, the presentation of portions of CT volume data may beutilized in other procedures besides orthopedic surgery of the hip,knee, and other joints. For example, a CT scan may be conducted of atumor. During a percutaneous biopsy of the tumor, images based on one ormore cut planes through the CT volume data may be generated andpresented to assist the surgeon in performing the biopsy.

Multiple AR Devices

In some embodiments, more than one person in the operating room 100 maybe wearing an AR device 200. For example, one or more assistants inaddition to the surgeon 114 may be wearing AR devices 200. The AR device200 worn by the surgeon may be primary AR device, which may operate as aserver, and the other AR devices may operate as clients of the primaryAR device.

FIG. 52 is a schematic illustration of an operating room 5200 inaccordance with one or more embodiments. Disposed in the operating room5200 is an operating table 5202 on which a patient 5204 is positionedfor a surgical procedure. A surgeon 5206 and at least one other medicalprofessional 5208 may be in the operating room 5200. The surgeon 5206and the medical professional 5208 may each be wearing an AR device 200 aand 200 b respectively. One or more of the AR devices, such as the ARdevice 200 a, may be connected to a server 5210 via a network 5212. Aphysical registration and tracking device 5214 may be docked to thepatient's pelvis. The AR devices 200 a and 200 b may present one or morevirtual images, e.g., holograms, during the surgical procedure on thepatient 5240. For example, a hologram 5216 of a cup impactor may bepresented in a planned location relative the patient's pelvis. Forexample, the AR device 200 a may detect the physical registration andtracking device 5214 and present the hologram 5216 of the cup impactor.The surgeon 5206 may guide a physical cup impactor 5218 to be alignedwith the hologram 5216 to achieve one or more goals of the surgicalprocedure, such as implanting a prosthetic cup component at a plannedlocation in the patient's pelvis. In some embodiments, one or more ofthe AR devices 200 a and 200 b may present a User Interface (UI), asindicated at 5220, in the operating room 5200, such as in space oragainst one or more walls of the operating room. The UI may be of aplanning application presenting a surgical plan for the surgicalprocedure on the patient.

Automated Object Recognition and Registration of Tools and BodyStructures.

The navigation system 1600 may receive data captured by one or more ofthe camera(s) on the AR device 200 of the surgical scene, such as imagedata in some embodiments. The object recognizer 1602 may detect anobject in the received image data, and the object tracker 1606 may trackthe detected object. For example, the AR device 200 may transmitcaptured image data, e.g., via the network device 112, to the dataprocessing device 100. The model database 1608 may be configured withdata regarding the shape of the patient-specific HipXpert tool, such asthree-dimensional (3D) shape for the HipXpert tool. As noted, the datamay be one or more CAD files, 3D model data, etc. The object recognizer1602 may search for an object in the received image data that matchesthis data, thereby identifying the HipXpert tool for example in theimage data. The information in the model database 1608 may include thedimensions of the HipXpert tool on a patient specific basis, e.g., asadjusted for a specific patient, and may also know the location of thepelvis relative to the HipXpert tool, for example as determined duringthe surgical planning phase. The object recognizer 1602 may detectand/or recognize the HipXpert tool in a field of view, e.g., the imagedata, and the object pose detector 1604 may determine its orientationfrom which the navigation system 1600 may then calculate and track thelocation of the patient's pelvis in space. The object recognizer 1602may implement the Vuforia Engine and Vuforia Model Targets technologyfrom PTC Inc. of Boston, Mass.

The surgeon may affix a second object, e.g., a tracker attached to thepatient's pelvis, that can then be tracked, and a calculation of thesecond object's location relative to the HipXpert tool can be made bythe navigation system 1600. The location of the pelvis can then bedetermined relative to this second object, allowing the HipXpert tool tobe removed. That is, the navigation system 1600 may recognize theHipXpert tool itself optically because its size and shape are known tothe system 1600, and so “seeing” it from any angle would allow for thedetermination of exactly where the HipXpert tool is positioned andoriented in space. The dimensions of the HipXpert tool and the predicteddocking of the HipXpert tool onto the patient's pelvis ispatient-specific, so the system 1600 may need to be configured withthose parameters on a patient-specific basis.

Other tools also can be tracked in space either by teaching the systemthe unique CAD geometry of the other tools or affixing an object that ismore easily tracked to the tool to be tracked. This may be useful for acup impactor or acetabular reamer. The same may be true for the femur orany instrument used on the femur. The femur may be registered byrecognizing a unique small visible section of the surface with a trackerattached to it, as described. The navigation system 1660 may track thefemur based on object recognition and tracking of the object. In someembodiments, a tracker may then be attached to the femur and trackingcontinued based on this tracker allowing the surgeon to change the femursurgically making it no longer recognizable while still allowing thefemur to be tracked. The process may be called patient-specific shaperecognition registration methodology.

As described, tracking may be performed using the spatial detectionsystem provided by the AR device 200, such as the depth camera 230 andthe IR emitters. For example, the navigation system 1600 may implementsimultaneous localization and mapping (SLAM) utilizing the datagenerated by the depth camera 230. In other embodiments, tracking may beperformed by two cameras of known relative orientation to allow forstereoscopic calculation. Further, the stereoscopic cameras could beaffixed to the AR device 200 as described, while in other embodimentsimage data from the 3D detection system 108 may be used by thenavigation system 1600 either alone or in combination with image datafrom the AR device 200. The advantage of acquiring the image informationfrom the one or more cameras on the AR device 200 is that the surgeonalways needs a primary line of site, giving the camera(s) of the ARdevice 200 the same line of site as the surgeon. This is in contrast tothe situation with traditional infrared stereoscopic cameras whereline-of-site competition between the surgeon and the camera can occur.The other advantage of having the camera(s) on the surgeon's head isthat the viewpoint of the camera(s) relative to the surgeon's eyes isknown so that an augmented reality display of virtual objects can bedisplayed in the same perspective that the real objects would be seen in(except that they would otherwise be invisible, being buried deep insidethe body) except perhaps for small exposed subsections during surgery.

In other embodiments, other tools besides by the HipXpert tool may beused and recognized and tracked by the navigation system 1600.

To aid in detecting a patient-specific object and determining itsorientation and/or pose, the object may be asymmetrical and/or uniquelyrecognizable within the surgical scene. For example, to the extent theobject is a tool, the tool may be asymmetrical. To the extent the objectis a body part, the body part may be asymmetrical. Nonetheless,symmetrical objects, such as body parts, and/or tools may be used in thepresent disclosure.

In some embodiments, the compass portion of the HipXpert device may beomitted or removed.

In some embodiments, a second object may be attached to the object,e.g., body part, or to the tool to aid in detecting the object or toolin the image data and/or in determining its orientation and pose. Thesecond object may be attached to the object or the tool in knowngeometric relationships such that locating the second object anddetermining its orientation and/or pose can be used to determine thelocation and/or orientation of the object and/or tool, e.g., using oneor more translations.

In further embodiments, one or more markings may be applied to theobject and/or tool to aid in its detection and/or in determining itsorientation and/or pose. For example, a checkerboard or other uniqueand/or recognizable pattern may be applied to the object.

During the planning stage, adjustments may be determined for thephysical registration and tracking tool 2100 so that it will fit, e.g.,be docked to the patient's pelvis, as planned. The adjustments mayinclude how far to slide out the extendable arms 2104 a and 2104 b sothat the tips of the legs contact the patient's pelvis at plannedlocations. Thus, the dimensions of the tool 2100 may vary from onepatient to another. Nonetheless, the dimensions of the hub 2102 of thetool 2100 is identical for all patients, e.g., it is a static componentof the tool 2100. Furthermore, as described, the cube 2108 may beattached to the hub 2102 of the tool 2100 in the same manner for allpatients.

In some embodiments, the cube 2108 with the QR code(s) may be omittedfrom the tool 2100. With this embodiment, the AR device 200 may beconfigured to recognize the physical tool 2100 in the operating room.For example, the AR device 200 may recognize one or more portions of thephysical tool 2100 that is the same for all patients, such as the hub2102. In this way, the same recognition process may be used for allpatients even though the tool 2100 also includes portions adjusted on apatient-specific basis, e.g., the extent to which the arms 2104 a and2104 b are extended. A patient-specific transformation matrix may bedetermined relative to the static portion of the tool being recognized,e.g., the hub 2102. Providing a portion of a registration and trackingtool that is static, e.g., the same, for all patients, and configuringthe AR device 200 to recognize this portion of the tool may be moreefficient, e.g., in terms of planning, processing and memory resources,than individually configuring the AR device 200 for each patient torecognize the tool as a whole as adjusted for each patient.

FIG. 45 is a perspective view of a hip registration and tracking tool4500. The tool 4500 may include an elongated support arm 4502, a supportframe 4510, a first moveable leg brace 4514, and a second moveable legbrace 4516. The elongated support arm 4502 may include a first end 4520.Disposed at the first end 4520 may be an opening 4522 configured toreceive an end of a first leg (not shown) that may extendperpendicularly from the support arm 4502. An end of a second leg may bereceived at the first moveable leg brace 4514, and an end of a third legmay be received at the second moveable leg brace 4516. The second andthird legs may also extend perpendicularly from the elongated supportarm 4502, like the first leg.

A first track 4534 may be formed along at least a portion of a frontside of the support arm 4502, and a second track (not shown) may beformed along at least a portion of a back side of the support arm 4502.The first and second tracks may be recessed tracks, such as slots orgrooves. The support frame 4510 may include a first edge that engagesthe first track 4534 securing the support frame 4510 to the elongatedsupport arm 4502, while allowing the support frame 4510 to slide alongthe front side of the elongated support arm 4502. The first moveable legbrace 4514, and thus the second leg, may be configured for slidableattachment to the back side of the elongated support arm 4502. Thesupport frame 4510 may include a second edge 4548 to which the secondmoveable leg brace 4516 may slidably attach.

The first leg may have a tip configured to contact the right ASIS.Second and third legs may be slidably attached to the elongated supportarm relative to the first leg. The distances between the first leg andthe second and third legs may be determined preoperatively so that, whenthe second and third legs, are set to these predetermined distancesalong the elongated support arm, a tip of the second leg contacts theleft ASIS, and a tip of the third leg contacts an anterior aspect of theischium of the patient's pelvis below the acetabulum of the hip beingoperated on. An operating surgeon may access the patient's hip jointusing the anterior approach or the anterolateral approach (e.g., withthe patient in the supine position), and may dock the apparatus to thepatient, thereby registering the patient's pelvis and establishing thepatient-specific, supine pelvic reference plane and/or coordinatesystem.

Mounted to the support frame 4510 may be a cube 4550 with one or more QRcodes. During surgery, the first moveable leg brace 4514 and the secondmoveable leg brace 4516 of the physical tool 4500 may be adjusted asplanned so that the tips of the respective legs contact the patient'spelvis at the planned locations. The tool 4500 may be docked to thepatient's pelvis. The AR device 200 may detect the one or more QR codeson the cube 4550 and may anchor one or more holograms as describedherein.

The tool 4500 may be flipped over so that it may be used to operate on apatient's left or right hips. The support frame 4510 and the cube 4550may also be flipped around so that it remains on top of the tool 4500.

Thus, the only things that may be specific for a patient when using aHipXpert registration and tracking tool or the tool 4500 are the armlengths or the positions of the moveable leg braces, respectively, andthe single patient-specific matrix, which may relate where therespective tool is in space to the raw image coordinate system from theCT scanner with the patient randomly placed within it.

In some embodiments, instead of utilizing a single tool that operates asa combination registration and tracking device, separate registrationand tracking tools may be utilized. For example, a cube with one or moreQR codes may be randomly attached to a patient's pelvis. A surgeon maythen register the patient's pelvis, e.g., utilize a digitizing probe todigitize a plurality of points on the patient's pelvis. The location ofthe cube with the one or more QR codes may then be determined relativeto the patient's pelvis as registered. The AR device 200 may thenpresent one or more holograms in the planned locations and as anchoredrelative to the cube with the one or more QR codes.

It should be understood that other elements besides or in addition to aQR code may be used to register the pelvis or another anatomicalstructure, such as a tracker.

FIG. 57 is a schematic illustration of a front view of a pelvis 5700 inaccordance with one or more embodiments. During the surgical procedure,a surgeon may attach a tracker 5702 to the pelvis 5700 at a randomlocation. In some embodiments, the AR device 200 may recognize thetracker 5702 by virtue of its shape using object recognition and/or theAR device 200 may recognize an image on the tracker 5702, such as by wayof example only a QR code. Alternatively, the tracker 5702 may includeoptical or magnetic elements that can be detected by the tracking system106. The surgeon may utilize a digitizing probe 5704 to digitize aplurality of points on the surface of the pelvis 5700. The AR device 200may similarly recognize the tracker using object and/or imagerecognition. Alternatively, the digitizing probe 5704 may includeoptical or magnetic elements that can be detected by the tracking system106. The navigation system 1600 may process the digitized points toregister the pelvis 5700. The navigation system 1600 may also track thepelvis 5700 via the tracker 5702 as detected by the AR device 200 or thetracking system 106. The AR device 200 may present one or more hologramsanchored to the pelvis 5700 relative to the tracker 5702.

It should be understood that a similar process may be used with otheranatomical structures, such as the knee.

Augmented Reality for Hip Replacement Surgery:

Having the navigation system 1600 know where the pelvis is and havingthe navigation system 1600 know where the display is located in front ofthe surgeon's eyes allows for the detailed display of virtual imagesincluding computer models, e.g., of the pelvis and one or more trackedtools, from the same perspective as the surgeon. This would allow thesurgeon to see the patient in reality, and also to see virtual objectssuch as the computer model of the pelvis projected onto the lenses ofthe AR device 200 in the same location as the actual object inside thepatient.

FIG. 4 is a pictorial representation of a surgical procedure showing aregistration tool, e.g., the HipXpert tool, docked on a particularpatient in accordance with one or more embodiments.

The location of the pelvis relative to the HipXpert tool may be knownpre-operatively, e.g., during a planning phase. Using the spatialdetection systems built into the AR device 200, the navigation system1600 can calculate the perspective of the 3D object, e.g., the HipXperttool, another tool, the patient's pelvis, another portion of thepatient's anatomy, etc., from the surgeon's viewing perspective at thatmoment.

FIG. 5 is an illustration of a 3D surface model of a pelvis with a modelof the registration and tracking device docked thereto in accordancewith one or more embodiments.

Having calculated the surgeon's perspective of the tool and the pelvis,a virtual model of the pelvis can then be projected onto the lenses ofthe AR device 200 and thus within the surgeon's point of view in realtime

FIG. 6 is a schematic illustration of an image projected by the ARdevice 200 showing a virtual image of the patient's pelvis underneaththe skin from the exact perspective of the surgeon at that moment inaccordance with one or more embodiments.

Similarly, tools that are used on the patient could be seen in realityand a superimposed virtual model of the same tool in the same locationcould be projected by the AR device 200 for viewing by the surgeon. Thiswould allow the surgeon to see the exact location of a part of the toolwhich, in reality, has disappeared inside of an incision, but yet avirtual image of which can be “seen” through the AR device 200.

Additionally, work that the tool accomplishes when being used can betracked by the navigation system 1600 and the object that is changed canbe updated. This would be true for example if a virtual display of thepelvis is projected as is a virtual display of an acetabular reamer. Thecamera(s) is able to track the relative locations of the two objects,and may also track and integrate an effect that the reamer has on theacetabulum, allowing for updating of the pelvis model to reflect theacetabular reaming itself and that could be compared both to theoriginal structure and the planned structure of the acetabulum that thesurgeon aims to achieve prior to implantation of the acetabular cupcomponent. Accordingly, the navigation system 1600 may show the surgeonwhere s/he started, where s/he are so far, and where s/he needs to gonext to accomplish to final goal of acetabular reaming.

Automated Object Recognition and Registration of Tools and BodyStructures: Example: A Small Field of View Inside the Acetabulum

An alternative method of calculating the location of the pelvis in realtime during total hip replacement surgery, for example is to get a smallview of the actual pelvis through the incision. Assuming the shape ofthe bone surface within that field of view is sufficiently unique, thenthe pelvis could be registered automatically by the navigation system1600 just by “seeing” a small part of this patient-specific, uniqueobject. For example, during total hip replacement, the femoral head isremoved and the inside of the acetabulum is exposed. As long as thespatial detection system can see this bony structure, an automated shaperegistration of the entire bone could be accomplished.

FIG. 7 is a pictorial representation of the view into the acetabulum ofa patient through an incision during surgery in accordance with one ormore embodiments.

FIG. 8 is an illustration of a 3D surface model of the patient's pelvisfrom the same perspective as FIG. 7 in accordance with one or moreembodiments. This matching registration can be done by the navigationsystem 1600, for example, by matching unique actual and virtual shapestogether using object recognition.

FIG. 9 is a schematic illustration of an image projected on the ARdevice 200 showing a virtual image of the patient's pelvis underneaththe skin from the same perspective of the surgeon at that moment inaccordance with one or more embodiments.

With existing systems, if instruments block the view or the bone surfaceis changed, then accurate registration and tracking is lost. Inaccordance with one or more embodiments of the present disclosure, thisdisadvantage can be avoided by attaching a separate tracker to the boneand transferring the relative information achieved through recognitionof the patient-specific object and then simultaneous identification ofthe location of the separate tracker to the pelvis. Then, so long as theseparate tracker can be tracked, surgery can proceed even though thesurface that was used to achieve initial registration has been modified.

The system could combine the registration techniques depicted in FIGS.4-6 and FIGS. 7-9 to achieve even greater accuracy.

Reality Feedback and Update Loop

In some embodiments, one or more anatomical structures may not beprepared in precisely the manner as planned. Nonetheless, a surgeon maydetermine that the partial preparation is acceptable, for example toachieve the one or more goals of the surgical procedure. For example,suppose a patient's acetabulum is prepared and a cup componentimplanted. However, suppose further that the cup component is notimplanted exactly as planned, e.g., the position and/or orientation ofthe cup component within the acetabulum is somewhat different than theplanned position and/or orientation. In some embodiments, the cameras orother sensors of the AR device 200 may be trained on the cup componentas implanted. The object recognizer 1602 may detect and recognize thecup component. The navigation system 1600 may determine the positionand/or orientation of the cup component as implanted and provide thisinformation to the surgical planning system 1700. The surgical planningsystem 1700 may update the surgical plan for the patient using theactual position and/or orientation of the cup component as implanted,rather than the planned position and/or orientation. In otherembodiments, the navigation system 1600 may determine the actualposition and/or orientation of the cup component as implanted bydetermining a final location of the cup impactor. For example, theobject recognizer 1602 may recognize the cup impactor while in its finallocation. The navigation system 1600 may determine the actual positionand/or orientation of the cup component based on the final location ofthe cup impactor and the known geometry of the acetabular liner that isthen inserted into the cup. For example, the navigation system 1600 maybe configured with the geometric relationship between the cup impactorand the cup component. Thus, the navigation system 1600 can derive theposition and/or orientation of the cup component from the positionand/or orientation of the cup impactor. Alternatively or additionally,one or more trackers may be attached to the cup impactor, and thenavigation system 1600 may determine the position and/or orientation ofthe cup impactor from the one or more trackers.

It should be understood that this is but one example of a realityfeedback and update mode of the present disclosure. Feedback andupdating the surgical plan may be performed with other elements besidesthe cup component and in other surgical procedures, such as knee repair.

In some embodiments, a sequence of holograms may be as follows:

1. pelvis and HipXpert device custom adjusted for the patient and dockedto patient's pelvis, with the pelvis unchanged;

2. pelvis and HipXpert device custom adjusted for the patient and dockedto patient's pelvis with the ideal cup bed as planned at the acetabulum;

3. HipXpert device custom adjusted for the patient (without pelvis),with ideal cup bed;

4-7. pelvis and HipXpert device custom adjusted for the patient anddocked to the patient's pelvis and with a sequence of reamers and reamerhandles in proposed locations. For example, if the planner wants to putin a 56 mm acetabular cup component, the planner might plan for the useof a 51 mm, a 53 mm, a 55 mm, and finally a 56 mm reamer. Each one ofthese reamers will do a certain amount of the work to achieve the finalcup bed at the acetabulum. Holograms could be generated for each reamerand, during surgery, the holograms could be presented and the surgeoncould work each reamer to match up with the hologram;

8. pelvis and HipXpert device custom adjusted for the patient and dockedto the patient's pelvis, the cup component and the cup impactor with thescrew holes of the cup component lined up in the planned orientation asthe cup can be rotated around the handle. Alternatively or in addition,a hologram of the cup component and the cup impactor floating in spaceso that the surgeon can line up the screw holes perfectly rotationally;

9. pelvis and HipXpert device custom adjusted for the patient and dockedto the patient's pelvis and the cup component and the cup impactor withthe cup component located at the final location. Then, during surgery,with the physical cup impactor that matches the hologram, the surgeonwould know that the cup component is in the planned, final location whenthe physical cup impactor and the physical cup component attachedthereto line up perfectly with the hologram;

10. pelvis and HipXpert device custom adjusted for the patient anddocked to the patient's pelvis and the cup component and the proposedscrews for the cup component with planned directions and lengths toindicate to the surgeon the planned, e.g., optimal, direction to drillin and how long the screws will be;

11a and b. pelvis and HipXpert device custom adjusted for the patientand docked to the patient's pelvis and cup component showing with (a)and without (b) surrounding osteophytes to show the surgeon what totrim. Having planned removal of osteophytes, the systems and methods candetermine what the potential impingement and/or free range of motionwould be from the surgery and could show this information, for examplebased on degree of osteophyte removal; and

12. pelvis and HipXpert device custom adjusted for the patient anddocked to the patient's pelvis and the cup component and the liner,e.g., the final product;

In some embodiments, the systems and methods may then do objectrecognition of the cup component and the pelvis to determine what theactual result of implantation is. Based on this information, the systemsand methods could recalculate impingement and/or range of motion, i.e.,on the spot, as desired.

Example of Clinical Implementation for Total Knee Arthroplasty.

Three technologies exist for Total Knee Arthroplasty (TKR). Theyinclude:

-   -   1. image-based registration and navigation (with or without        robotics) and/or statistical shape modeling, e.g., based on a        large data set and patient-specific characteristics;    -   2. image-free registration and navigation (with or without        robotics); and    -   3. physical template registration.

Image-based navigation of TKR was one of the first methods employedwhere 3D models and coordinate systems are developed in advance, andthen the bones are registered in surgery by digitizing surface pointsthat allow for matching registration. This method fell out of favoruntil a more recent resurgence with robotics.

Alternatively, image-free methods allow for knee navigation by (with atracker affixed) moving the hip around to triangulate its position,directly digitizing ankle points, and then putting in the requisiteinformation on the distal femur and proximal tibia with a digitizer.

A third method (which is image-based) makes a physical template thatlocks into the anatomy in a predictable way. The physical template maycontain cutting slots to allow for bone surface resection as planned.Alternatively, the template may be used to transfer the information toone or more other tools, for example by having drill holes for thedrilling of holes within the bone for the placement of pins. Thetemplate may then be removed and another surgical tool that fits overthe same pins in a predictable way may be affixed and used. Used thisway, these physical templates do not allow for the traditionalnavigational calculations such as alignment, ligament balance and rangeof motion but they do allow for accomplishing the goals of the surgeryin a more basic way.

Again, alternatively, the physical templates may be used as aregistration and tracking device for subsequent navigation. An exemplaryphysical template is the acetabular template disclosed in U.S. Pat. No.8,986,309 for an Acetabular Template Component and Method of Using SameDuring Hip Arthroplasty, which is hereby incorporated by reference inits entirety.

FIG. 44 is a partial side view of a patient's pelvis 4402 showing thepatient's acetabulum 4404 and acetabulum rim 4406 with a custom fittedtemplate 4408 in accordance with one or more embodiments. The customfitted template 4408 may be generally circular shaped to mate with allor a substantial portion of the patient's acetabular rim 4406. Becausethe template 4408 matches the rough and uneven shape of the acetabularrim 4406, it fits to the rim 4406 and thus the pelvis in a singleorientation. The template 4408 may have an upper surface 4414 and alower surface 4420 opposite the upper surface 4414. Mounted on the uppersurface 4414 may be a cube 4430 having QR codes (not shown) on at leastsome of its surfaces or faces. The lower surface 4420 is shaped to matchthe acetabular rim 4406. The template 4408 may have an open interior4418 so that the template 4408 does not interfere with the placement ofan acetabular cup component within the patient's acetabulum 4404.

The template 4408 may be held in place by one or more fasteners, such asscrews 4422. With the template 4408 fitted to the patient's acetabulum,the AR device 200 may detect one or more of the QR codes on the cube4430 and register the patient's pelvis. One or more patient-specifictransformation matrices may be associated with the cube 4430 and/or QRcodes and used to determine the orientation and position of virtualimages, e.g., holograms, relative to a QR code and/or the cube 4430.

Automated Object Recognition and Registration of Tools and BodyStructures: Example: The Distal Femur for TKR

The present disclosure may use the spatial detection system of anaugmented reality HMD for example to register and track anatomicalstructures and/or tools, for example by recognizing the threedimensional orientation of a portion of exposed anatomy, e.g., as viewedthrough an incision. For example, the knee may be opened and the spatialdetection system or the camera(s) in the AR device 200 may see the endof the femur. The navigation system 1600 may then track the orientationof the entire femur by having the one or more of the sensors or camerasof the AR device 200 see a portion of the patient-specific anatomicalobject. In some embodiments, this may be referred to as an object-based,image-based methodology in which a particular object is identifiedpre-operatively and the navigation system 1600 searches the image datafor that particular patient-specific object in the operative scene. Asnoted, for hip surgery, the HipXpert tool is tuned, e.g., adjusted, tothe particular patient, and the navigation system 1600 is prepared torecognize that the HipXpert tool as adjusted for the patient within theimage data of the surgical scene. Based on the detection of the patientspecific object within the surgical scene, the navigation system 1600may then register the rest of the “internal” scene, e.g., the patient'spelvis, another anatomical component or feature, etc.

For total knee replacement (TKR), CT, MR, statistical shape or otherpredictive modeling, or other data may be obtained of the patient'sfemur, tibia, hip, and ankle pre-operatively. The acquired data may beused to generate 3D models, which may be in the form of CAD files, ofpatient's femur, tibia, hip, and ankle, including the portions of thefemur and tibia that are to be exposed during TKR surgery. These modelsmay be stored in the model database 1608 of the navigation system 1600and utilized during the object recognition and object orientation/posedetermination steps.

In some embodiments, the AR device 200 may perform object recognition ofthe top end of the tibia. Suppose, the top surface of the tibia isamorphous such that the AR device 200 locks on the location of the tibiajust with object recognition of the top of the tibia leading toinsufficient registration. The object recognition may be sufficient forheight of the tibial articular surface for example, but not for accuracyof the longitudinal axis. The AR device 200 may present a hologram ofthe whole tibia and a QR cube on the tibia with a phase 1 registrationstep of less than sufficient registration based on object recognition ofthe proximal tibia alone. If the AR device 200 presents the tibia—topand bottom, as a hologram, and the tracker keeps the top end closelymatched, then the surgeon could move the patient's ankle into positionto make it coincident with the hologram. That is, the surgeon may movethe reality, e.g., the patient's leg, into alignment with the augmentedreality, e.g., the hologram.

In the case of the femur for knee replacement, the surgeon may want tomodify the very anatomical part that the navigation system 1600 istracking, which might otherwise end the tracking. To obviate this, thenavigation system 1600 may transfer the information to an object, suchas a tracking frame, affixed to the bone that could still be trackedthroughout the procedure. Using this technique, the navigation system1600 can recognize and track the entire femur by seeing a sufficientamount of the distal femur and matching it up in real time to a virtualmodel of the entire femur. If a tracking frame is then attached, itsrelationship to the model of the entire femur can then be determined bythe navigation system 1600, e.g., through one or more geometrictranslation operations. In the case of patient-specific anatomicalobject recognition and registration of the bone that is being describednow, the location of the bone is already known at the time that thetracking frame is affixed so there is no subsequent registration step.Once the information it transferred to the second object, the bone canthen be modified in the surgery and tracked throughout the procedureeven though the original patient-specific anatomical object that wasinitially used to determine the location of the bone no longer exists ina subsequent stage of the surgery. FIG. 10 is a pictorial representationof a patient's knee showing a view of the distal femur during total kneereplacement in accordance with one or more embodiments.

In some embodiments, a physical template having a surface that matchesthe surface of the distal femur or the proximal tibia may be attached tothe femur or tibia. The physical template may include a tracker. Thetracking system 106 and/or the AR device 200 may recognize the physicaltemplate and/or the tracker and register the femur or tibia. A trackermay be attached to the femur or tibia in a random manner, and theregistration of the femur or tibia may be transferred to this tracker.The physical template may then be removed and the procedure continued.

FIG. 11 is an illustration of a 3D surface model of the patient's femurintended to depict the exact same bone in the exact same orientation asthe surgeon's view as determined by automated patient specificanatomical object recognition in accordance with one or moreembodiments.

FIG. 12 is a schematic illustration of an image projected by the ARdevice 200 showing a virtual model of the femur placed in space in theexact same place as the actual femur as seen from the surgeon's point ofview in accordance with one or more embodiments.

Again, attaching a tracker to the bone would allow the registrationinformation to be transferred to the tracker so that the surfaces thatwere originally used to achieve registration can be modified. This wouldallow for continued navigation and augmented reality displaycontinuously from the surgeon's point of view no matter what that viewis. In addition to trackers having optical or magnetic elements, such asthe tracker 5702 illustrated in FIG. 57 , a tracker may be a 2D or 3Dshape that is spatially unique and thus recognizable by the AR device200. Exemplary 3D shapes include an optical tracker without thereflective elements, e.g., just the arm elements. Exemplary 2D shapesinclude a metal plate having a non-symmetrical star shape or anon-symmetrical cross shape, etc.

As disclosed, in some embodiments, the present disclosure may replacethe use of physical templates, such as templates used at the knee and/oracetabulum. Instead, the system effectively presents a virtual template,such as a hologram of a template, that locks onto the patient's anatomyusing patient-specific anatomical object recognition instead of anactual 3D printed physical template. The navigation system 1600 maynavigate knee surgery instruments using one or more QR codes and/orobject recognition. For example, the sequence may start with resectionof the distal femur. In this case, the process may proceed as follows:

-   -   1. A QR cube may be affixed to the femur by the surgeon.    -   2. The AR device 200 may recognize the distal femur using object        recognition, thus preliminarily registering the femur. The AR        device 200 may also track the QR cube. Registration of the femur        may be augmented by moving the hip around, watching the QR cube        and/or distal femoral object, and calculating the hip center.        For example, the AR device 200 may track the QR cube to        calculate the hip center and may also or alternatively use        object recognition to track the location of the distal femur        during motion to triangulate to the hip joint center. This may        be done before the QR cube or other tracker is affixed, but is        preferably done after the tracker is attached. It should be        understood that another tracking device, besides the QR cube may        be used. In fact, it could just be another unique “object” that        could also be tracked using object recognition as opposed to        image recognition.    -   3. Once the femur is registered and holograms may be anchored,        the AR device 200 may present one or more holograms of an ideal        distal femoral cut plane. The surgeon may then put any distal        cutting block in the field. In fact, a metal sheet may be placed        into the saw blade slot of the cutting block, and the surgeon        may place the cutting jig so that the metal sheet is coincident        with the hologram of the distal cut plane hologram. The surgeon        may then pin the jig in place, do the cut, and compare the cut        to the hologram of the cut. The surgeon could then fine tune the        cut, either through the jig or free-hand. Before projecting a        hologram of the 4 in 1 cutting block, the AR device 200 may        project a hologram of the proposed drill holes that would be        needed for the pegs on the back of the 4 in 1 cutting block.    -   4. This process also may be applied to cutting blocks that        provide multiple cutting planes, such as “4 in 1” cutting        blocks. Typically, the 4 in 1 cutting block is affixed to the        femur with two pegs or pins. The AR device 200 may display a        hologram of the preferred locations of the pin hole, as planned,        and the surgeon may drill holes to match the hologram, and put        on the 4 in 1 cutting jig. Then, the AR device 200 may display a        hologram of the preferred, e.g., planned, distal femoral        preparation surface including the anterior, anterior bevel,        distal, posterior bevel, and posterior cuts. These could be        visually checked by the surgeon before making the cuts and again        a metal sheet could be placed in the cutting slots to assist in        lining up the physical 4 in 1 cutting block to the hologram        thereof. Then, after the cuts are made and the 4 in 1 cutting        block (or other jig) is removed, the AR device 200 may display a        hologram of the cut surfaces as planned, and the surgeon can        again fine tune the bone cuts either through the jig or free        hand again to match the cut surfaces presented in the hologram.        -   The AR device 200 may project the ligament distraction with            a hologram perpendicular to the tibia for the surgeon to            check ligament balance and possibly change rotation of the 4            in 1 cutting block before completing this step.    -   5. At this point, the distal femoral preparation is complete.        However, the AR device 200 can also display a hologram that        shows the bone and the final femoral component on it.    -   6. The AR device 200 can register the tibia through object        recognition of the exposed proximal tibial bone surface. In some        embodiments, a tracker, such as a QR cube, may be placed on the        tibia, e.g., to improve registration. The AR device 200 may        present a hologram of the initial registration, and then        rotating this hologram around manually, e.g., through user        interaction by the surgeon, or automatically using the surgeon's        palpation of the medial and lateral malleoli of the ankle. This        process could replace traditional registration with image-free        navigation of the knee, in which a tracker is put on the tibia        and points are digitized on the proximal tibia. In addition,        with traditional registration, the tip of the digitizer is        placed on the skin compressed on the medial malleolus and then a        second point is acquired with the tip of the digitizer placed on        the skin compressed on the lateral malleolus. The traditional        registration provides information on the longitudinal axis of        the tibia. With the present technique, the longitudinal axis is        included in the hologram. The surgeon puts a finger and thumb on        the medial and lateral malleoli, and the hologram is then        rotated to be placed between the finger and the thumb. In this        way, the tibia is registered by moving the virtual axis in line        with the ankle distally. This technique obviates the need for a        digitizer. It may also leverage the capability of the AR device        200, which monitors the surgeon's hand and could be used to        automatically determine the location of the surgeon's thumb and        finger at the same time or allow a single finger to be the        digitizer of the two ankle points. In some embodiments, a        digitizer with a QR cube on it could be used to register the        tibia. In other embodiments, object recognition of the specific        digitizing object may be used to register the tibia. In sum, the        systems could use various combinations of object recognition of        the proximal tibia, direct digitization (which may require        tracking two objects), tracking an object without a tracker on        it such as the tip of the surgeon's finger or a standard,        uniquely shaped digitizing instrument (the tip of which could be        tracked by tracking the whole object using object recognition).        Such an object-recognition-tracked-digitizer could be used to        help with femoral registration as well. No QR cube needs to be        included on the digitizer.    -   7. Once the tibia is registered and displayed in one or more        holograms, the planned proximal tibial cut plane may be        displayed in one or more holograms. In addition, the AR device        200 can also display in one or more holograms not only the        tibia, but a model of whatever extramedually resection cutting        jib is to be used. For simplicity's sake, if the AR device 200        displays just the cut plane, then the surgeon can put any        cutting jig against the tibia, put a physical metal cut plane        saw blade replacement through the tibial cut plane saw blade        cutting slot, and then the instrument could be pinned to the        tibia at that location, where the physical representation of the        proposed cut plane matches with the hologram of the proposed cut        plane. The cut could then be made and then the surgeon could        compare the achieved cut plane to the planned cut plan as        displayed by the hologram.    -   8. Additional tibial holograms that may be displayed by the AR        device 200 include showing drill pin holes for placement of the        tibial preparation tray that determines rotation and a keel for        the tibial component. Another hologram that may be displayed by        the AR device 200 may show the tibia and the tibial metal        component with the proposed plastic insert within, i.e., the        final implant appearance as planned.    -   9. A common method of determining femoral AP position and        rotation is anatomically using the posterior femoral condyles.        The posterior cut plane is typically a predetermined (e.g.,        9 mm) distance from the backs of the two condyles and        perpendicular to the distal cut plan. The anterior cut plane is        then a fixed distance from the posterior and purely dependent        upon the size of the proposed component. A surgeon, however,        before just going with an anatomical measure, may check the        ligament balance. This can be done by registering the tibia        before the femoral preparation is finished. The tibial resection        guide may be placed against the femur with the ligaments between        the femur and tibia distracted by retractors. The surgeon can        then visually check that the tibial jig is parallel to the        proposed rotation of the femur (based on anatomical landmarks).        This would show that the ligament balance would be good if the        bone cuts are performed where proposed anatomically. In some        embodiments, a similar process may be followed without the        tibial jig, for example by displaying a hologram with a plane        that is perpendicular to the long axis of the tibia onto the        femur. With the ligaments distracted, it should match up with        the pin holes of the 4 in 1 cutting jig. This can be checked by        the surgeon before the femur is prepared. If the two methods        (anatomic and ligament distraction) do not agree, the surgeon        has a choice of releasing ligaments, changing the femoral        rotation, or a combination of the two.    -   11. In some embodiments, the AR device 200 may display holograms        of one or more Anterior Posterior (AP) cutting jigs used in knee        surgery. The cutting jigs may be patient-specific or their        locations recommended on a patient-specific basis and include        indications of where bone cuts are to be made. During surgery,        the AR device 200 may display the holograms at a planned        location manually, automatically, e.g., using QR codes or object        recognition, or a combination of manually and automatically.

The system may apply a similar technique to the tibia or any other bodypart internal or external.

FIG. 13 is a pictorial representation of a patient's knee showing thetibia during total knee replacement in accordance with one or moreembodiments.

FIG. 14 is an illustration of a 3D surface model of the patient's tibiaintended to depict the exact same bone in the exact same orientation asthe surgeon's view as determined by automated surface matching usingstereoscopic cameras or any other method of stereoscopic surfacedetection in accordance with one or more embodiments.

FIG. 15 is a schematic illustration of a hologram projected by the ARdevice 200 showing a virtual model of the tibia placed in space in theexact same place as the actual tibia as seen from the surgeon's point ofview in accordance with one or more embodiments.

Patient-specific anatomical object recognition and CAD file automatedsurface matching registration methodology may replace use of a physicaltemplate. The CAD file of the patient specific anatomical object to berecognized may be prepared pre-operatively with the object thenrecognized in surgery by searching the data provided by the spatialdetection system of the AR device 200 to determine and track thelocation of the object.

The object may also be tracked either directly or indirectly, e.g.,through another object associated with the primary object, such as atracker placed on the pelvis or the femur, among other options. Again,the tracking may be performed by the spatial detection system (e.g.,cameras and/or other sensors) on the AR device 200, the tracking system106, or the 3D detection system 108, among others. The presentdisclosure may also eliminate having to make and sterilize a physicaltemplate and instead could be planned immediately. The presentdisclosure may eliminate extensive digitization of surfaces that mightotherwise be necessary for image-free or image-based knee navigation.

Combinations of registration techniques (such as digitizing the anklelandmarks or triangulating the center of rotation of the hip joint)could be employed to improve accuracy further.

As noted, knee arthroplasty procedures generally require resection orcutting of both the patient's femur at its distal end and the patient'stibia at its proximal end. These resections or cuts are conventionallyaccomplished with the aid of cutting jigs or blocks that are placed onthe respective bones and guide and direct the surgeon in the cutting ofthe bones at a desired location and orientation. In some embodiments,the cutting jigs or blocks may be patient-specific.

FEMUR DISTAL RESECTION. In a planning stage, a 3D model of apatient-specific distal femoral cutting jig or block may be created,e.g., based on a 3D model of the patient's femur. The location of themodel of the femoral cutting jig or block on the femur may be planned sothat the distal end of the femur will be cut as planned. The model ofthe femoral cutting jig or block may be used to generate a hologram forpresentation by the AR device 200. During surgery, an anatomicalstructure or a tracker may be recognized, the AR device 200 may presentthe hologram of the femoral cutting block at the planned location at thedistal end of the femur. In some embodiments, the surgeon may thenco-locate the physical cutting block with the hologram, and secure thephysical cutting block in place. The surgeon may then utilize thephysical cutting block to resect the distal end of the patient's femur.

In some embodiments, the AR device 200 may present a hologram of anideal cut plane so that the surgeon could double check the cut planecreated by performing a bone preparation cut as guided by the physicalcutting jig or block. In addition, with this embodiment, the surgeon mayfree-hand fine tune the resection of the distal end of the femur to moreperfectly match the planned resection in the case that the cut thatoccurred through the cutting jig or block was close but not perfect.

TIBIAL RESECTION. As with the femur, a 3D model of a patient-specificproximal tibial cutting jig or block may be created, e.g., based on a 3Dmodel of the patient's tibia. The location of the model of the tibialcutting jig or block on the tibia may be planned so that the proximalend of the tibia will be cut as planned. The model of the tibial cuttingjig or block may be used to generate a hologram for presentation by theAR device 200. During surgery, an anatomical structure or a tracker maybe recognized, the AR device 200 may present the hologram of the tibialcutting jig or block at the planned location at the proximal end of thetibia. In some embodiments, the surgeon may then co-locate the physicalcutting block with the hologram, and secure the physical cutting blockin place. The surgeon may then utilize the physical cutting block toresect the proximal end of the patient's tibia. Similarly as with thefemur, the AR device 200 may present a hologram of an ideal cut plane.

Several methods may be used to plan the resections of the femur andtibia, including the ideal cut planes, and thus where to place thecutting jigs or blocks.

Method 1. Pure Anatomy. A basic way to determine the placement of the APcutting jig is preoperatively, based purely on preop imaging. Manyvendors of physical templates utilize this method. While this method iseasy, it does not take ligament balance into consideration.

Method 2. Pure Ligament Balance. Known as the “Insall Technique” for Dr.John Insall. With this method, the distal femoral and proximal tibialresections are made more or less orthogonal to the long axis of theirrespective axes with minor variations depending upon surgicalphilosophy, but the rotation of the anterior and posterior cuts can bedone with many different philosophies. In addition to a purely anatomicdetermination based on preop imaging (or intraop digitization), theopposite philosophy would be by ligament distraction technique. This isa classical method. Suppose that the tibia cut is more or less square toits long axis. Suppose further that the surgeon wants the back of thefemur to be parallel to that so that when the knee is bent 90 degrees,that the back of the femur and the top of the tibia are parallel (withthe ligaments distracted at the time of determination). This means thatwhen the surgeon puts the implants in, that the ligament tension withthe knee bent 90 degrees is more or less even. This method involves kneebalancing using component rotation and/or soft tissue release asvariables at the surgeon's disposal to accomplish this task.

Method 3. Blended Technique. With this method, a surgeon may look atfemoral rotation using both methods 1 and 2 and see if they agree. Ifthey do not, the surgeon may do slightly more ligament releasing to makethe two methods a little closer to each other. This blended methodbasically moves method 2 closer to method 1.

Any of these methods may be implemented by the present disclosure.

In some embodiments, one or more guides may be used to determine whereto place the cutting blocks or jigs. For example, the LEGION total kneesystem from Smith & Nephew Inc. of Memphis, Tenn. includes a sizingguide that uses the posterior femora condyles as a reference. The sizingguide may be used by the surgeon to determine where to place a cuttingblock or jig. In some embodiments, the same sizing guide may be used tocorrectly place a range of cutting blocks or jigs.

FIG. 31 is a front view of a sizing guide 3100 having two locator holes3102 a and 3102 b in accordance with one or more embodiments.

FIG. 32 is a perspective view of the sizing guide 3100 on a femur 3202in accordance with one or more embodiments.

With the sizing guide 3100 attached to the patient's femur, a surgeonmay utilize the locator holes 3102 a and 3102 b to drill two holes intothe patient's femur. Next, the sizing guide 3100 may be removed and acutting block or jig may be attached to the femur using the two holesdetermined by the locator holes 3102 a and 3102 b of the sizing guide3100. The cutting block or guide may define correct Anterior, Posterior,and angled chamfer cuts for the implant.

FIG. 33 is a front view of a cutting block 3300 attached to a patient'sfemur 3302 in accordance with one or more embodiments. The cutting block3300 may be attached to the patient's femur 3302 using pins or screwsextending into drill holes 3304 a and 3304 b that were formed using thelocator holes 3102 a and 3102 b of the sizing guide 3100.

FIG. 34 is a side view of the cutting block 3300 attached to thepatient's femur 3302 in accordance with one or more embodiments. Thecutting block 3300 provides an Anterior cutting guide 3402, a Posteriorcutting guide 3405, and two angled chamfer cutting guides 3403 and 3404.A surgeon may place a saw blade 3406 in the cutting guides, e.g., theAnterior cutting guide 3402, of the cutting block 3300 to make theplanned cuts.

In some embodiments, the AR device 200 may present holograms of theplanned locations of the locator pin holes. During surgery, theseholograms may be presented and surgeon may use the holograms to align adrill to drill the locator pin holes at the planned locations. Forexample, a hologram of the sizing guide having drill holes may bepresented by the AR device 200. In this implementation, the surgeon maynot use the physical sizing guide or any other guide. After drilling thelocator pin holes based on the one or more holograms presented by the ARdevice 200, the surgeon may install the cutting block or guide on thepatient's femur. In some embodiments, the AR device 200 may presentholograms of the planned Anterior, Posterior, and angled chamfer cuttingplanes as planned. The surgeon may then check that these holograms ofthe cutting planes are co-located, e.g., aligned, with the cuttingguides of the physical cutting block or jig.

After making the cuts, the surgeon may implant a prosthetic kneecomponent.

FIG. 35 is a perspective view of a prosthetic knee component 3500 inaccordance with one or more embodiments. The knee component 3500 mayinclude interior surfaces that match the Anterior, Posterior, and angledchamfer cuts made to the patient's femur. In addition, the kneecomponent 3500 may include two pins 3502 a and 3502 b that may bereceived in the drill holes that were formed based on the hologram ofthe locator pin holes.

It should be understood that other guides and/or cutting blocks may beused, such as a five-in-one cutting block or jig, among others.

Other Knee Solutions

It should be understood that other procedures may be utilized.

Pre-operative imaging of the patient's knee may be performed using CT,MR, or other imaging techniques. Alternatively, statistical shapedmodels having minimal patient-specific information for model fitting maybe used.

Software steps may include segmentation of femur and tibia. Landmarksand coordinate systems may then be created. For example, the femoralcoordinate system rotation could initially be determined by theposterior femoral condyles.

Next, a plan for the femoral component may be created with the followinginitial criteria:

-   -   1. Distal femoral cut perpendicular to the long axis of the        femur in the coronal plane and in a few degrees of flexing in        the sagittal plane.    -   2. The distal cut plane is to be Xmm (e.g., 8 mm or 9 mm) from        the most distal cartilage surface of the femur.    -   3. The posterior and anterior cut planes of the femur are        planned such that the posterior cut plane is Xmm anterior (e.g.,        8 or 9 mm) anterior to the posterior femoral condyles and        perpendicular to the distal femoral resection (unless the        particular implant system calls for a slight angle). The        anterior cut may be parallel to the posterior cut and determined        by the size of the planned femoral component which is in turn        determined by the size of the femur.

A plan for the tibial component may be created with the followinginitial criteria:

-   -   1. The proximal tibial resection plane may be perpendicular to        the long axis of the tibia on the coronal plane and typically in        a few degrees of flexion in the sagittal plane. One way to        determine the rotation of the tibial coordinate system, since        the knee may have been in extension during the imaging, is to        just project the femoral condylar rotation onto the tibia.        Another way is to use one or two more anatomical points in        addition to the point where the tibial long axis exits the        proximal tibia. In addition, the depth of the resection plane        below the tibial surface could be either Xmm below the lowest        tibial plateau surface or Xmm below the highest tibial plateau        surface. This could be a surgeon preference variable.

The AR device 200 may be configured to present the following sequence ofstatic holograms:

F01 The native femur.

F02. The native femur with the distal femoral cut plane.

F03. The native femur, distal femoral cut plane, and generic or impactspecific traditional distal femoral resection guide.

F04. Two drill hole holograms that would project into the distal femurthat would tell the surgeon where to drill holes so that when a specific“4 in 1” femoral cutting jig is placed using the locating pins on theback surface of jig, the anterior and posterior bone cuts (and chamfercuts) made using the jig are in the planned locations.

F05. Hologram of the preferred prepared surface of the distal femur thathas modified the femur to reflect the distal, anterior, posterior, andchamfer bone cuts.

F06. A hologram of the femur with the femoral component of the plannedsize in the planned place.

INSALL-technique holograms. The above describes preparing the femurbased purely on anatomical landmarks. In some embodiments, ligamentbalance may be assessed in surgery. One or more holograms may bepre-generated and pulled from a patient specific holographic library forthe surgery during ligament balance.

Suppose, the AR device 200 is tracking the tibia and displayingholograms relative to the tibia. Suppose further that a determination ismade regarding where the proximal tibial resection will be. Assume thefemoral and tibial implant thicknesses taken together are X mm (e.g., 5mm above the low side of the tibial surface and another 9 mm for theposterior portion of the femoral implant). The surgeon may distract theligaments and the AR device 200 may project a hologram of a cut planerelative to the tibia in that location. The hologram may be projectedupon the femur. This hologram projection may be compared to where thepin holes and posterior cut plane suggested by anatomical landmarkingwould be.

If the system projected the hologram based on tracking the tibia andalso projected a hologram simultaneously based on tracking the femur,these two holograms should ideally overlap. To the extent they differ,the surgeon has several choices:

-   -   A. the surgeon may stick with the anatomical position and        rotation based in the femoral anatomy, ignoring the tibial        information,    -   B. the surgeon may use the tibial/ligament distraction        recommendation,    -   C. the surgeon may do more ligament releasing to get the two        holograms to line up more closely, and    -   D. with or without C, the surgeon may choose a femoral position        somewhere between A and B.

The AR device 200 may present the following tibial holograms:

T01. A hologram of the native tibia.

T02. A hologram of the native tibia plus the preferred tibial cut plane.

T03. T02 plus a proximal tibial cutting jig, either generic or vendorsystem specific.

T04. A hologram projected on the cut tibial surface that has drill holeprojections onto the tibial surface marking where the drill holes wouldgo for the tibial tray jig that is affixed to the tibia in the correctposition and rotation and for allowing the Keel Punch preparation to bein the correct place.

T05. A hologram of the tibia reflecting the ideal tibial resection plusthe planned tibial component in the correct position and rotation.

Registration and Tracking.

For the femur, a tracking object, such as a cube with one or more QRcodes, may be attached in a predetermined location. The AR device 200may then register the femur primarily using object recognition of theunique patient specific distal femoral anatomy. Registration of thefemur may be augmented by other classical methods, such as kinematictriangulation of the femoral head center, direct landmark digitizationthrough the incision, or even REVERSE REGISTRATION, in which the ARdevice 200 may project a hologram and the surgeon may move the limb intoposition to overlap the hologram. The AR device 200 may then anchor thehologram at this location.

The tibia could be registered using the reverse registrationmethodology, since the proximal tibial surface is less distinct in itsunique geometric characteristics than the distal femur.

When trial or real implants are in place, the AR device 200 can trackboth the femur and tibia and project above described holograms T05 andF06, e.g., in real time, as the knee is moved about. The AR device 200may also calculate alignment and motion and ligament balance.

ACL Reconstruction.

Anterior cruciate ligament (ACL) reconstruction is the most common majornon-prosthetic reconstruction procedure on the knee. The current stateof the art is to perform arthroscopy and to:

1. debride the stumps of the ruptured ACL,

2. prepare the “notch” of the femur,

3. use anatomic landmarks to determine the femoral and tibial attachmentpoints of the new ligament,

4. place a drill hole through the tibia with the hole in the joint atthe proposed tibial attachment point,

5. place a drill hole into the femur with the hole location in the jointat the proposed femoral attachment point,

6. thread the ligament through,

7. secure the femoral attachment using an interference screw (for abone-ligament-bone graft),

8. tension the ligament, and

9. secure the tibial attachment, again using an interference screw (fora bone-ligament-bone graft).

The procedure is slightly different if a soft tissue graft is used sinceit is tied down with other fixation methods.

Some of the disadvantages of the current state of the art include:

1. performance of the procedure requires constant visualization withinthe joint and any clouding of the fluid with bleeding or debris preventsthat visualization.

2. determination of the “isometric points” for the tibia and femoralattachments of the graft is not scientifically determined and placingthe graft in the wrong location can lead to early rupture.

As described herein, the system of the present disclosure may implementthe following technique:

1. optionally, the AR device 200 could be used as both the tracking andvisualization technology.

2. Preoperative CT/MR or predictive shape modeling. Create 3D models for“virtual template” object recognition registration in surgery.

3. in surgery, attach tracking objects to the femur and tibiapercutaneously that can be recognized and tracked by the spatialdetection system of an augmented reality HMD.

4. utilize an arthroscopy camera that has: a) stereoscopic vision or asimilar spatial detection system inside the joint at the working end andb) a tracking object on the end that is outside of the body that can betracked by the spatial detection system of the AR device (or any othertracking method). The technique may utilize two spatial detectionsystems: one within the joint that can be used to visualize the anatomyfor “virtual template” registration and the second being, for example onthe AR device 200, that can watch the position of the external end ofthe arthroscopy device.

5. for femoral registration, the spatial detection system on thearthroscopy device is aimed at the femoral joint surface or any aspectof the femur visible within the joint. The edges of the joint surface,for example, may be used in object detection. With the external spatialdetection system watching the position of the arthroscopy device, theknee is moved through a range of motion to be able to identify enough ofthe unique CAD surface of the femur to determine where the femur is inspace relative to the tracker previously placed on the femur. As in theother examples, a patient-specific CAD or other image file may becreated, such as a CAD file of the surface of the patient's femur or aportion thereof, such as the distal end of the patient's femur.Furthermore, a patient-specific coordinate system may be determinedpre-operatively relative to this object, e.g., the patient's femur. Thenavigation system 1600 may then search for and detect this object in theimage data obtained from the spatial detection system on the AR device200 of the surgical scene. Once detected, the navigation system 1600 mayalso register the actual object, e.g., based on the pre-operativelydetermined, patient-specific coordinate system. The navigation system1600 would then register the location of the entire femur in space.

6. for tibial registration, the spatial detection system on thearthroscopic device may be aimed at the tibial joint surface. With theexternal spatial detection system (or other tracking solution), thetibia can be registered. Here, the process includes a preoperative CADfile of the tibial surface based on MRI, and the navigation system 1600can recognize the exact location of that surface using the internalarthroscopic spatial detection system with the external reference frameattached. The endoscope may be moved to capture more of the surface,such as the surface of the tibia.

7. Proposed attachment points on the tibial and femoral surface may beplanned based on anatomy on the MR. Alternatively, after registration,the knee may be cycled through the range of motion in an “ACL competent”position (pressure tensioning the PCL during motion). In this way, thesystem may calculate optimal isometric points on the femur and tibia.

8. Now that the system knows where the femur and tibia are at all timesand where the ideal attachment points are as the ligaments attach in theknee, the system can display the femur and tibia on the AR device 200from whatever viewpoint the surgeon 114 has at that moment. The systemcan also show the proposed course of tibial and femoral tunnels foroptimal ACL reconstruction. In addition, the system can track any toolsincluding traditional tunnel creating instruments and show the tools inaugmented reality and show the virtual project of the proposed tunnelrelative to those tools.

At least some of the advantages of the present disclosure include: ACLreconstruction can be performed more reliably since the attachmentpoints would be more reliably placed, reducing the risk of ACLreconstruction failure. The methodology has the further benefit thatmost of the critical parts of the procedure can be done with augmentedreality, reducing the need for arthroscopic visualization, allowing forrefining the surgery for visualization with arthroscopy just at specificpoints during the procedure instead of continuously. In addition tobetter technical excellence, with proper refinement, the surgerypotentially could be performed more efficiently.

Dental Surgery.

Dentists generally place dental implants without navigation or enhancedvisualization of any kind. Dentists may rely on plain radiographs and/orCT imaging with multiplanar reformatting, which can show where theavailable bone is upon which to base a dental implant. However, problemscan arise if the bone is quite thin, which can occur particularly on thebuccal side of the maxilla or mandible or when teeth have been missingfor some time, and if landmarks such as adjacent teeth are not availableto spatially guide the dentist. Traditional image-based navigation israrely used in this field for reasons such as complexity, cost, and thefact that the patient is in one place, and the navigation information isin another, such as on an LCD screen.

Holographic guidance during dental implant surgery may represent asignificant and cost-effect advance in this field. The anchoring ofholographic guidance may be based at least in part on one or moreexisting teeth, which are physically available as opposed to beingdeeply under the skin.

In some embodiments, the present disclosure may include:

-   -   1. Create a dental mold on at least a portion of a patient's        mandible or maxilla, depending on which side of the jaw needs an        implant. The mold may include any available teeth that would        allow attachment of a tracker that could be outside of the        mouth. Such a tracker could be a traditional dynamic reference        base (DRB) for infrared (IR) stereoscopy tracking, or in some        embodiments a QR code or other recognizable image or object that        can be identified and continuously tracked by the navigation        system 1600. The location of the mold may be adjacent to the        location of the proposed dental implant or other proposed        procedure while spaced far enough away to still provide access        to the area at which the implant or other dental surgery will be        located. In some embodiments, the mold may include an element        from which the location of the tracker can be determined. The        element may be the tracker itself, a tracker support, a tracker        attachment mechanism, or information, such as dots or dimples on        the mold, from which the location of the tracker may be        determined.    -   2. With the mold in place on the patient's jaw, obtain a CT or        other imaging study of the patient prior to the procedure. The        CT or other imaging includes the element or information from        which the location of the tracker may be determined. For        example, in some embodiments, the tracker may be attached to the        mold when the CT or other imaging is performed. In other        embodiments, the tracker support or tracker attachment mechanism        may be included with the mold, but the tracker itself may be        omitted, when the CT or other imaging is performed.    -   3. The surgical planning system 1700 can use the CT or other        imaging to plan the location of the implant and to identify        exactly where the tracker, e.g., the QR code or other tracking        image or object will be located within the CT coordinate space.        For example, a computer-generated 3D model of the patient's        mandible or maxilla including the mold and tracker may be        generated. A planner may select one or more particular tools,        e.g., drills, drill bits, etc., and/or implants and determine        locations of the surgical tools and/or implants relative to the        3D model. For example, 3D models of the surgical tools and/or        implants may be combined with the 3D model of the mandible of        maxilla to form new 3D models.    -   4. Having the tracker or a mold to which the tracker will be        attached on the patient before the imaging may simplify the        patient registration process, for example because the tracker        location is already known. This is in contrast to registering        the pelvis for hip surgery where the CT study is performed        without a tracker affixed and then the location of the pelvis is        determined during the surgery, such as by docking a registration        and tracking tool to the pelvis or affixing a tracker then        registering the location of the pelvis relative to the tracker        subsequently.    -   5. With the tracker affixed to the patient in a known way, the        surgical planning system 1700 can utilize combinations of the 3D        models to generate one or more static and/or dynamic holograms        for display by the AR device 200. In some embodiments, at least        some of these holograms may show the patient anatomy otherwise        not visible to the dentist and be co-located with actual patient        anatomy, e.g., that may be visible to the dentist and thus the        AR device 200.    -   6. Exemplary holograms may include one or more holograms of the        patient's mandible or maxilla without the implant and one or        more holograms of the patient's mandible or maxilla with the        surgical tools and/or the implant at the planned locations. By        presenting such holograms using the AR device 200, the dentist        can “see” exactly in 3D where the patient's bone is located to        properly anchor an implant.    -   7. During the surgical procedure, the mold and tracker may be        inserted in the patient's mouth. The AR device 200 as worn by        the dentist may recognize the tracker and may present and anchor        the one or more holograms in space relative to the tracker so        that the holograms of the mandible or maxilla are co-located        with the patient's physical mandible and maxilla and the        holograms of the surgical tools and/or implant are presented at        their planned locations.    -   8. The dentist may utilize the one or more holograms as guides        in operating the surgical instruments and implanting the        implant.    -   9. In some embodiments, the AR device 200 worn by the dentist        may present the CT data volume for the patient's mandible or        maxilla co-located with the patient's mandible or maxilla during        the procedure. For example, the AR device 200 may generate one        or more planar cuts, e.g., cut planes, through the CT data        volume to produce a two dimensional (2D) CT image from the CT        data. The AR device 200 may present this 2D CT image to the        dentist. By co-locating the CT data volume with the patient, the        2D CT image, as displayed by the AR device 200, may appear to        the dentist as overlaid on and co-located with the patient's        anatomy. The cut plane may be set at a predetermined distance        from the AR device 200.    -   10. It should be understood that the surgical planning system        1700 may generate other, e.g., more sophisticated, holograms        such as one showing the exact trajectory of a planned drill        hole, or the exact size and location of the implant itself.        Furthermore, the navigation system 1600 and the surgical        planning system 1700 may update the holograms, e.g., in real        time from the perspective of the dentist, to show the effect of        any tools that have been used up until that point in the        procedure.

One advantage of such a methodology is that the entire planning andnavigation process may be performed on the spot in a single sessionwhere the custom dental mold for the tracker is affixed to one or moreof the patient's available teeth, the imaging can then take place, theplanning and holograms can be quickly performed and generated, and theimplant intervention can then take place. Another advantage is that thetracker/dental mold apparatus may be removed and replaced onto thepatient that day or another day with the reapplication of the trackergoing back to the same exact place that it was previously. Accordingly,any number of procedures may be performed on the same day or subsequentdays using the same planning and registration or updated planning withthe same registration. A third advantage is that this methodology andtechnology is inexpensive.

FIG. 56 is a top view of an example dental model 5600 in accordance withone or more embodiments. The dental mold 5600 includes an impression5602 of a patient's teeth. The dental mold 5600 also includes aprojection 5604. Disposed on the projection 5604 is a pattern indicatedat 5606. The pattern 5606 is formed from a plurality, e.g., three,markings 5608 a-c. During the surgical procedure, the AR device 200 mayrecognize the pattern 5606 and may anchor one or more holograms relativeto the pattern 5606.

FIG. 53 is an illustration of a planning window 5300 in accordance withone or more embodiments. The planning window 5300 includes a 3D model ofa patient's mandible 5302 and a 3D model of a tracker 5304 attached to a3D model of a tracker support 5306. The tracker support 5306 may beattached to a dental mold 5308 on the patient's mandible 5302. Locationsof one or more surgical instruments and one or more implants may beplanned to achieve one or more goals. For example, a model of an implant5310 may be placed at the mandible 5302 at a planned location.Alternatively or additionally, a drill axis 5312 for drilling into themandible 5302 to receive the implant may be planned at a preoperativelydetermined location. One or more holograms may be generated from themodels created in the surgical plan and the holograms may be presentedby the AR device 200 and anchored relative to the tracker 5304.

FIG. 54 is an illustration 5400 of cut planes that may be presented bythe AR device 200 during a surgical procedure in accordance with one ormore embodiments. The illustration includes a first pane 5402 that showsa 3D surface model of the 5404 of at least a portion of a patient's jawincluding the patient's mandible 5406. Superimposed on the 3D surfacemodel 5404 are three boxes illustrating orthogonal cut planes. A red box5408 signifies one image-generation plane, a yellow box 5410 signifies asecond image-generation plane, and a green box 5412 signifies a thirdimage-generation plane. The illustration 5400 includes additional panespresenting cut planes through CT volume data of the patientcorresponding to the planes of the boxes 5408, 5410 and 5412, which maybe presented by the AR device 200 in the exact location within thepatient's jaw. For example, a pane 5414 illustrates a cut plane throughthe CT volume data for the red box 5408. A pane 5416 illustrates a cutplane through the CT volume data for the yellow box 5410. A pane 5418illustrates a cut plane through the CT volume data for the green box5412.

Neurosurgery/Ear Nose Throat (ENT) Surgery

Holographic guidance during neurosurgery and/or ENT surgery also mayrepresent a significant and cost-effect advance in this field. As withthe dental surgery embodiment, the holographic guidance may be based atleast in part on one or more existing teeth. For example, a dental moldfor example of a patient's upper teeth (as they are fixed relative tothe patient's skull) may be made and a tracking object such as a QR codemay be affixed to the dental mold. Imaging may then take place, duringwhich the tracking object (e.g., QR code) may be identified on theimages, and then the rest of the procedure may be planned. Thisembodiment may use the upper teeth and a prior-to-imaging application ofa tracker that can then be included in the preop plan so thatregistration may be instant, automatic, and accurate.

It should be understood that additional and/or alternative registrationtechniques may be used. To improve the registration accuracy, the“virtual template” registration method may be combined with othermethods. For the femur, triangulation of the center of rotation of thehip can be calculated by moving the hip around with a stereoscopiccamera tracking the tracker attached to the femur. Combining this withthe virtual template registration could further refine the accuracy ofregistration. Combining digitization with the virtual templateregistration could further refine the accuracy of registration.

The foregoing description of embodiments is intended to provideillustration and description, but is not intended to be exhaustive or tolimit the disclosure to the precise form disclosed. Modifications andvariations are possible in light of the above teachings or may beacquired from a practice of the disclosure. For example, while a seriesof acts has been described above with respect to the flow diagrams, theorder of the acts may be modified in other implementations. In addition,the acts, operations, and steps may be performed by additional or othermodules or entities, which may be combined or separated to form othermodules or entities. Further, non-dependent acts may be performed inparallel.

Further, certain embodiments of the disclosure may be implemented aslogic that performs one or more functions. This logic may behardware-based, software-based, or a combination of hardware-based andsoftware-based. Some or all of the logic may be stored in one or moretangible non-transitory computer-readable storage media and may includecomputer-executable instructions that may be executed by a computer ordata processing system. The computer-executable instructions may includeinstructions that implement one or more embodiments of the disclosure.The tangible non-transitory computer-readable storage media may bevolatile or non-volatile and may include, for example, flash memories,dynamic memories, removable disks, and non-removable disks.

No element, act, or instruction used herein should be construed ascritical or essential to the disclosure unless explicitly described assuch. Also, as used herein, the article “a” is intended to include oneor more items. Where only one item is intended, the term “one” orsimilar language is used. Further, the phrase “based on” is intended tomean “based, at least in part, on” unless explicitly stated otherwise.

The foregoing description has been directed to specific embodiments ofthe present disclosure. It will be apparent, however, that othervariations and modifications may be made to the described embodiments,with the attainment of some or all of their advantages. Therefore, it isthe object of the appended claims to cover all such variations andmodifications as come within the true spirit and scope of thedisclosure.

What is claimed is:
 1. A system comprising: a registration and trackingdevice configured to dock to a portion of a patient's pelvis in apredetermined and fixed location; a computer-based surgical planningsystem configured to: present a two-dimensional (2D) or athree-dimensional (3D) model of the portion of the patient's pelvis;determine a location for a 3D model of the registration and trackingdevice as docked to the 2D or 3D model of the portion of the patient'spelvis; establish a coordinate system for the registration and trackingdevice; determine a location of one or more surgical tools relative tothe coordinate system for the registration and tracking device; generateone or more files from which a plurality of holograms may be produced ofcombinations of two or more of: the 2D or 3D model of the portion of thepatient's pelvis; the registration and tracking device; and the one ormore surgical tools; and an augmented reality (AR) head-mounted device(HMD), the AR HMD including: at least one sensor configured to recognizeat least a portion of the registration and tracking device; one or moreprojectors configured to present the plurality of holograms; and anavigation system that tracks the registration and tracking device andanchors the plurality of holograms in a space based on the coordinatesystem for the registration and tracking device.
 2. The system of claim1, wherein the computer-based surgical planning system is furtherconfigured to determine a location of at least one implant relative tothe coordinate system for the registration and tracking device and theplurality of holograms further include the at least one implant.
 3. Thesystem of claim 1, wherein the registration and tracking device includesa three dimensional (3D) shape having a surface and one or more markingsis disposed on the surface of the 3D shape, and the at least one sensorof the AR HMD recognizes the one or more markings on the surface of the3D shape of the registration and tracking device.
 4. The system of claim3 wherein the one or more markings is one or more Quick Response (QR)codes or a checkerboard pattern.
 5. The system of claim 1 wherein theregistration and tracking device includes a hub having a predeterminedand fixed shape and three legs extending from the hub, the three legsconfigured to dock the registration and tracking device to the portionof the patient's pelvis, and the at least one sensor of the AR HMDrecognizes the hub of the registration and tracking device.
 6. Thesystem of claim 1 wherein the computer-based surgical planning system isfurther configured to: determine a location of the one or more surgicaltools relative to a coordinate system for the 2D or 3D model of theportion of the patient's pelvis; and generate one or more transformationmatrices between the coordinate system for the 2D or 3D model of theportion of the patient's pelvis and the coordinate system for theregistration and tracking device, and wherein the AR HMD utilizes theone or more transformation matrices to anchor the plurality of hologramsin the space.
 7. The system of claim 1 wherein the computer-basedsurgical planning system is further configured to determine a sequenceof presentation for the plurality of holograms and the AR HMD presentsthe plurality of holograms in the sequence of presentation.
 8. Acomputer-implemented method comprising: presenting a two-dimensional(2D) or a three-dimensional (3D) model of a portion of a patient'spelvis; determining a location of a registration and tracking device asdocked to the 2D or 3D model of the portion of the patient's pelvis;establishing a coordinate system for the registration and trackingdevice; determining locations of one or more surgical tools and at leastone implant relative to a coordinate system for the 2D or 3D model ofthe portion of the patient's pelvis; generating files for presentingholograms of the one or more surgical tools and the at least one implantat the determined locations relative to the coordinate system for the 2Dor 3D model of the portion of the patient's pelvis; generating atransformation matrix between the coordinate system for the 2D or 3Dmodel of the portion of the patient's pelvis and the coordinate systemfor the registration and tracking device; and exporting the files to anaugmented reality (AR) head-mounted device (HMD).
 9. Acomputer-implemented method comprising: recognizing a registration andtracking device as docked to a portion of a patient's pelvis, therecognizing including tracking a location of the registration andtracking device; receiving files for presenting holograms of one or moresurgical tools and at least one implant at determined locations relativeto a coordinate system for the patient's pelvis; receiving atransformation matrix determining orientations and positions of theholograms relative to a coordinate system for the registration andtracking device; and utilizing the transformation matrix to present theholograms anchored at the determined locations.
 10. Thecomputer-implemented method of claim 9 wherein the registration andtracking device includes a hub having a predetermined and fixed shapeand three legs extending from the hub, the three legs configured to dockthe registration and tracking device to the portion of the patient'spelvis, and the recognizing the registration and tracking deviceincludes recognizing the hub of the registration and tracking device.11. The computer-implemented method of claim 9 wherein the registrationand tracking device includes a three dimensional (3D) shape having asurface and one or more markings is disposed on the surface of the 3Dshape, and the recognizing the registration and tracking device includesrecognizing the one or more markings on the surface of the 3D shape ofthe registration and tracking device.
 12. The computer-implementedmethod of claim 11 wherein the one or more markings is one or more QuickResponse (QR) codes or a checkerboard pattern.
 13. A system comprising:a computer-based surgical planning system configured to: present atwo-dimensional (2D) or a three-dimensional (3D) model of a patient'sknee; establish a coordinate system for the patient's knee; determine alocation of one or more cut planes to the patient's knee relative to thecoordinate system for the patient's knee; determine a location of atleast one implant for the patient's knee relative to the coordinatesystem for the patient's knee; generate one or more files from which aplurality of holograms may be produced of combinations of two or moreof: the 2D or 3D model of the patient's knee; the one or more cut planesto the patient's knee; and the at least one implant for the patient'sknee; and an augmented reality (AR) head-mounted device (HMD), the ARHMD including: at least one sensor configured to recognize at least aportion of the patient's knee, the recognize including tracking the atleast a portion of the patient's knee; one or more projectors configuredto present the plurality of holograms; and a navigation system thatanchors the plurality of holograms in a space based on the coordinatesystem for the patient's knee.
 14. The system of claim 13 wherein the atleast a portion of the patient's knee recognized by the at least onesensor of the AR HMD is a portion of a femur as exposed during asurgical procedure or a portion of a tibia as exposed during thesurgical procedure.
 15. The system of claim 14 wherein the navigationsystem of the AR HMD registers the at least a portion of the patient'sknee based on the recognize the at least a portion of the patient's kneeand transfers registration of the at least a portion of the patient'sknee to a tracker affixed to the patient's knee.
 16. The system of claim13 wherein the one or more cut planes include an anterior cut plane, ananterior bevel cut plane, a posterior cut plane, and a posterior bevelcut plane.
 17. A system comprising: a computer-based surgical planningsystem configured to: present a two-dimensional (2D) or athree-dimensional (3D) model of at least a portion of a patient'smandible and/or maxilla including a dental mold attached to one or moreof the patient's teeth, the dental mold including an element thatdefines a predetermined location of a tracker; establish a coordinatesystem for the tracker; determine a location of at least one dentalimplant in the portion of the patient's mandible and/or maxilla relativeto the coordinate system for the tracker; generate one or more filesfrom which one or more holograms may be produced of: the 2D or 3D modelof the portion of the patient's mandible and/or maxilla; and the atleast one dental implant; and an augmented reality (AR) head-mounteddevice (HMD), the AR HMD including: at least one sensor configured torecognize the tracker attached to the dental mold attached to the one ormore of the patient's teeth; one or more projectors configured topresent the one or more holograms; and a navigation system that anchorsthe one or more holograms in a space based on the coordinate system forthe tracker.
 18. The system of claim 17 wherein the element that definesthe predetermined location of the tracker is: the tracker; a support forthe tracker; an attachment mechanism for the tracker; or informationincorporated in the dental mold.
 19. The system of claim 17 wherein thetracker is at least one of a dynamic reference base, one or more QuickResponse codes, a recognizable image, or a recognizable object.